[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260806 20:33:22.002857 13282 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.248.190:33869
I20260806 20:33:22.005088 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:33:22.006645 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:22.033913 13291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:22.033932 13288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:22.034121 13289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:22.035086 13282 server_base.cc:1034] running on GCE node
I20260806 20:33:23.290591 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:33:23.290855 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:23.291006 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048403290993 us; error 0 us; skew 500 ppm
I20260806 20:33:23.292162 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:23.305387 13282 webserver.cc:460] Webserver started at http://127.12.248.190:38165/ using document root <none> and password file <none>
I20260806 20:33:23.306608 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:23.306802 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:23.307348 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:23.314844 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/master-0-root/instance:
uuid: "aad7d81155684a0c9b005ce25ca89e20"
format_stamp: "Formatted at 2026-08-06 20:33:23 on dist-test-slave-10pc"
I20260806 20:33:23.324653 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20260806 20:33:23.331797 13297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:23.333395 13282 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:33:23.333861 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/master-0-root
uuid: "aad7d81155684a0c9b005ce25ca89e20"
format_stamp: "Formatted at 2026-08-06 20:33:23 on dist-test-slave-10pc"
I20260806 20:33:23.334300 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:23.385564 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:23.387398 13282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:33:23.387934 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:23.468114 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.190:33869
I20260806 20:33:23.468200 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.190:33869 every 8 connection(s)
I20260806 20:33:23.478677 13352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:23.505806 13352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20: Bootstrap starting.
I20260806 20:33:23.513229 13352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:23.515497 13352 log.cc:824] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20: Log is configured to *not* fsync() on all Append() calls
I20260806 20:33:23.520682 13352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20: No bootstrap required, opened a new log
I20260806 20:33:23.544878 13352 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7d81155684a0c9b005ce25ca89e20" member_type: VOTER }
I20260806 20:33:23.545558 13352 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:23.545814 13352 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad7d81155684a0c9b005ce25ca89e20, State: Initialized, Role: FOLLOWER
I20260806 20:33:23.546883 13352 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7d81155684a0c9b005ce25ca89e20" member_type: VOTER }
I20260806 20:33:23.547446 13352 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:33:23.547727 13352 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:33:23.548090 13352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:23.552109 13352 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7d81155684a0c9b005ce25ca89e20" member_type: VOTER }
I20260806 20:33:23.553135 13352 leader_election.cc:302] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad7d81155684a0c9b005ce25ca89e20; no voters:
I20260806 20:33:23.555043 13352 leader_election.cc:288] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:33:23.555467 13355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:23.558102 13355 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 1 LEADER]: Becoming Leader. State: Replica: aad7d81155684a0c9b005ce25ca89e20, State: Running, Role: LEADER
I20260806 20:33:23.559152 13355 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7d81155684a0c9b005ce25ca89e20" member_type: VOTER }
I20260806 20:33:23.560426 13352 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:33:23.573213 13356 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad7d81155684a0c9b005ce25ca89e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7d81155684a0c9b005ce25ca89e20" member_type: VOTER } }
I20260806 20:33:23.573953 13356 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [sys.catalog]: This master's current role is: LEADER
I20260806 20:33:23.575188 13357 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad7d81155684a0c9b005ce25ca89e20. Latest consensus state: current_term: 1 leader_uuid: "aad7d81155684a0c9b005ce25ca89e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7d81155684a0c9b005ce25ca89e20" member_type: VOTER } }
I20260806 20:33:23.575960 13357 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [sys.catalog]: This master's current role is: LEADER
I20260806 20:33:23.577188 13365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:33:23.594508 13365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:33:23.596829 13282 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 20:33:23.614943 13365 catalog_manager.cc:1347] Generated new cluster ID: 4d4b87b550b24a1c90be3f4d4eb18158
I20260806 20:33:23.615301 13365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:33:23.632207 13365 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:33:23.634270 13365 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:33:23.651604 13365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20: Generated new TSK 0
I20260806 20:33:23.652835 13365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:33:23.663793 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:23.673444 13374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:23.674679 13375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:23.679344 13377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:23.679674 13282 server_base.cc:1034] running on GCE node
I20260806 20:33:23.680552 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:33:23.680858 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:23.681110 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048403681092 us; error 0 us; skew 500 ppm
I20260806 20:33:23.681831 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:23.686570 13282 webserver.cc:460] Webserver started at http://127.12.248.129:38437/ using document root <none> and password file <none>
I20260806 20:33:23.687122 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:23.687319 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:23.687608 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:23.688939 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-0-root/instance:
uuid: "6430856814214be2be901e26e0c9ae73"
format_stamp: "Formatted at 2026-08-06 20:33:23 on dist-test-slave-10pc"
I20260806 20:33:23.694957 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 20:33:23.699501 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:23.700393 13282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:33:23.700764 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-0-root
uuid: "6430856814214be2be901e26e0c9ae73"
format_stamp: "Formatted at 2026-08-06 20:33:23 on dist-test-slave-10pc"
I20260806 20:33:23.701045 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:23.725297 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:23.726643 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:23.729735 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:33:23.734875 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:33:23.735113 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:23.735419 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:33:23.735580 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:23.851853 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.129:45137
I20260806 20:33:23.851946 13447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.129:45137 every 8 connection(s)
I20260806 20:33:23.859794 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:23.873288 13454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:23.879856 13282 server_base.cc:1034] running on GCE node
W20260806 20:33:23.880975 13456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:23.881767 13452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:23.883122 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:33:23.883360 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:23.883555 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048403883541 us; error 0 us; skew 500 ppm
I20260806 20:33:23.884151 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:23.887624 13282 webserver.cc:460] Webserver started at http://127.12.248.130:37717/ using document root <none> and password file <none>
I20260806 20:33:23.888273 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:23.888557 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:23.888999 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:23.890399 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-1-root/instance:
uuid: "9819c221b89b4b3c829980ffc65b3679"
format_stamp: "Formatted at 2026-08-06 20:33:23 on dist-test-slave-10pc"
I20260806 20:33:23.893356 13448 heartbeater.cc:347] Connected to a master server at 127.12.248.190:33869
I20260806 20:33:23.893829 13448 heartbeater.cc:464] Registering TS with master...
I20260806 20:33:23.895004 13448 heartbeater.cc:511] Master 127.12.248.190:33869 requested a full tablet report, sending...
I20260806 20:33:23.898132 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 20:33:23.899930 13315 ts_manager.cc:194] Registered new tserver with Master: 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137)
I20260806 20:33:23.903321 13461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:23.903954 13315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40424
I20260806 20:33:23.904312 13282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 20:33:23.904667 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-1-root
uuid: "9819c221b89b4b3c829980ffc65b3679"
format_stamp: "Formatted at 2026-08-06 20:33:23 on dist-test-slave-10pc"
I20260806 20:33:23.904965 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:23.921873 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:23.923117 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:23.925753 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:33:23.928717 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:33:23.928929 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:23.929180 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:33:23.929354 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:24.037856 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.130:46879
I20260806 20:33:24.038038 13527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.130:46879 every 8 connection(s)
I20260806 20:33:24.050505 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:24.059877 13531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:24.065121 13532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:24.068001 13528 heartbeater.cc:347] Connected to a master server at 127.12.248.190:33869
I20260806 20:33:24.068467 13528 heartbeater.cc:464] Registering TS with master...
I20260806 20:33:24.069433 13528 heartbeater.cc:511] Master 127.12.248.190:33869 requested a full tablet report, sending...
I20260806 20:33:24.069829 13282 server_base.cc:1034] running on GCE node
W20260806 20:33:24.071235 13534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:24.071900 13315 ts_manager.cc:194] Registered new tserver with Master: 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:24.072358 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:33:24.072645 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:24.072840 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048404072824 us; error 0 us; skew 500 ppm
I20260806 20:33:24.073550 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:24.074002 13315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40438
I20260806 20:33:24.077315 13282 webserver.cc:460] Webserver started at http://127.12.248.131:35441/ using document root <none> and password file <none>
I20260806 20:33:24.077857 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:24.078087 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:24.078397 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:24.079540 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-2-root/instance:
uuid: "038a7b6ab86f465884b0861c91d19d36"
format_stamp: "Formatted at 2026-08-06 20:33:24 on dist-test-slave-10pc"
I20260806 20:33:24.084275 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260806 20:33:24.087760 13539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:24.088652 13282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:33:24.088935 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-2-root
uuid: "038a7b6ab86f465884b0861c91d19d36"
format_stamp: "Formatted at 2026-08-06 20:33:24 on dist-test-slave-10pc"
I20260806 20:33:24.089270 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786048401793185-13282-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:24.113895 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:24.115643 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:24.117288 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:33:24.120738 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:33:24.120965 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:24.121209 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:33:24.121376 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:24.223551 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.131:37989
I20260806 20:33:24.224108 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.131:37989 every 8 connection(s)
I20260806 20:33:24.249027 13604 heartbeater.cc:347] Connected to a master server at 127.12.248.190:33869
I20260806 20:33:24.249500 13604 heartbeater.cc:464] Registering TS with master...
I20260806 20:33:24.250491 13604 heartbeater.cc:511] Master 127.12.248.190:33869 requested a full tablet report, sending...
I20260806 20:33:24.252801 13315 ts_manager.cc:194] Registered new tserver with Master: 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:24.253762 13282 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.024508749s
I20260806 20:33:24.254977 13315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40444
I20260806 20:33:24.258054 13282 test_util.cc:268] Using random seed: -860881684
I20260806 20:33:24.313927 13315 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40458:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:33:24.318526 13315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:33:24.446120 13411 tablet_service.cc:1467] Processing CreateTablet for tablet e0a9ed1966eb4cc6968fb4e1b419e47a (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 20:33:24.446863 13491 tablet_service.cc:1467] Processing CreateTablet for tablet e0a9ed1966eb4cc6968fb4e1b419e47a (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 20:33:24.461741 13411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a9ed1966eb4cc6968fb4e1b419e47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.464582 13491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a9ed1966eb4cc6968fb4e1b419e47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.465911 13487 tablet_service.cc:1467] Processing CreateTablet for tablet dcbd7ac041ae408bbac21f2768bf5279 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 20:33:24.465788 13485 tablet_service.cc:1467] Processing CreateTablet for tablet 647a1ce4dc1c4b08b438a2a46d716cb6 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 20:33:24.469635 13489 tablet_service.cc:1467] Processing CreateTablet for tablet f25ceb6a22484feb825ea23218a15ac5 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 20:33:24.464843 13408 tablet_service.cc:1467] Processing CreateTablet for tablet dcbd7ac041ae408bbac21f2768bf5279 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 20:33:24.474658 13484 tablet_service.cc:1467] Processing CreateTablet for tablet e2cfa0f2971548ca8daf530a5b7d6e93 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 20:33:24.465832 13482 tablet_service.cc:1467] Processing CreateTablet for tablet 42329950ce5e4e14baf56b8caa9710f1 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 20:33:24.472512 13409 tablet_service.cc:1467] Processing CreateTablet for tablet 04a9c71cc716412683f508e467bb11e0 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 20:33:24.474746 13412 tablet_service.cc:1467] Processing CreateTablet for tablet c6eae037fbe24283b5b6bfe3a6c9c699 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 20:33:24.479103 13409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a9c71cc716412683f508e467bb11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.476999 13403 tablet_service.cc:1467] Processing CreateTablet for tablet 42329950ce5e4e14baf56b8caa9710f1 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 20:33:24.476018 13404 tablet_service.cc:1467] Processing CreateTablet for tablet e2cfa0f2971548ca8daf530a5b7d6e93 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 20:33:24.483704 13412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6eae037fbe24283b5b6bfe3a6c9c699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.476837 13488 tablet_service.cc:1467] Processing CreateTablet for tablet 04a9c71cc716412683f508e467bb11e0 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 20:33:24.486398 13487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbd7ac041ae408bbac21f2768bf5279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.484498 13492 tablet_service.cc:1467] Processing CreateTablet for tablet c6eae037fbe24283b5b6bfe3a6c9c699 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 20:33:24.485220 13405 tablet_service.cc:1467] Processing CreateTablet for tablet 647a1ce4dc1c4b08b438a2a46d716cb6 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 20:33:24.487449 13410 tablet_service.cc:1467] Processing CreateTablet for tablet f25ceb6a22484feb825ea23218a15ac5 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 20:33:24.485332 13568 tablet_service.cc:1467] Processing CreateTablet for tablet c6eae037fbe24283b5b6bfe3a6c9c699 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 20:33:24.483538 13406 tablet_service.cc:1467] Processing CreateTablet for tablet f36040d25e4a4103a76ea1099f0c25e8 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 20:33:24.493513 13485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647a1ce4dc1c4b08b438a2a46d716cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.483690 13486 tablet_service.cc:1467] Processing CreateTablet for tablet f36040d25e4a4103a76ea1099f0c25e8 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 20:33:24.488190 13567 tablet_service.cc:1467] Processing CreateTablet for tablet e0a9ed1966eb4cc6968fb4e1b419e47a (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 20:33:24.497411 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42329950ce5e4e14baf56b8caa9710f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.500602 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6eae037fbe24283b5b6bfe3a6c9c699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.502609 13566 tablet_service.cc:1467] Processing CreateTablet for tablet f25ceb6a22484feb825ea23218a15ac5 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 20:33:24.504658 13562 tablet_service.cc:1467] Processing CreateTablet for tablet f36040d25e4a4103a76ea1099f0c25e8 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 20:33:24.496588 13565 tablet_service.cc:1467] Processing CreateTablet for tablet 04a9c71cc716412683f508e467bb11e0 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 20:33:24.507936 13567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a9ed1966eb4cc6968fb4e1b419e47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.508821 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 647a1ce4dc1c4b08b438a2a46d716cb6 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 20:33:24.510108 13410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25ceb6a22484feb825ea23218a15ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.512060 13406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36040d25e4a4103a76ea1099f0c25e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.514745 13560 tablet_service.cc:1467] Processing CreateTablet for tablet e2cfa0f2971548ca8daf530a5b7d6e93 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 20:33:24.503257 13563 tablet_service.cc:1467] Processing CreateTablet for tablet dcbd7ac041ae408bbac21f2768bf5279 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 20:33:24.516666 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 42329950ce5e4e14baf56b8caa9710f1 (DEFAULT_TABLE table=test-workload [id=926776812466483f9ad678ec3a6ef0bd]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 20:33:24.519886 13484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2cfa0f2971548ca8daf530a5b7d6e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.521747 13405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647a1ce4dc1c4b08b438a2a46d716cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.525198 13482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42329950ce5e4e14baf56b8caa9710f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.526718 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25ceb6a22484feb825ea23218a15ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.535928 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647a1ce4dc1c4b08b438a2a46d716cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.534054 13486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36040d25e4a4103a76ea1099f0c25e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.553109 13408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbd7ac041ae408bbac21f2768bf5279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.568929 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2cfa0f2971548ca8daf530a5b7d6e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.579172 13492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6eae037fbe24283b5b6bfe3a6c9c699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.585469 13563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbd7ac041ae408bbac21f2768bf5279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.588634 13489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25ceb6a22484feb825ea23218a15ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.593739 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2cfa0f2971548ca8daf530a5b7d6e93. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:33:24.604825 13364 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:33:24.606703 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42329950ce5e4e14baf56b8caa9710f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.609129 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36040d25e4a4103a76ea1099f0c25e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.616568 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a9c71cc716412683f508e467bb11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.615826 13488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a9c71cc716412683f508e467bb11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:24.831189 13623 tablet_bootstrap.cc:492] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:24.841341 13622 tablet_bootstrap.cc:492] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:24.853649 13623 tablet_bootstrap.cc:654] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:24.884208 13624 tablet_bootstrap.cc:492] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:24.909278 13448 heartbeater.cc:503] Master 127.12.248.190:33869 was elected leader, sending a full tablet report...
I20260806 20:33:24.955451 13624 tablet_bootstrap.cc:654] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:24.960253 13622 tablet_bootstrap.cc:654] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:24.965989 13624 tablet_bootstrap.cc:492] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:24.966450 13624 ts_tablet_manager.cc:1397] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.125s user 0.008s sys 0.006s
I20260806 20:33:24.968816 13623 tablet_bootstrap.cc:492] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:24.968689 13624 raft_consensus.cc:348] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:24.969269 13623 ts_tablet_manager.cc:1397] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.148s user 0.013s sys 0.009s
I20260806 20:33:24.969434 13624 raft_consensus.cc:374] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:24.969789 13624 raft_consensus.cc:729] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:24.970539 13624 consensus_queue.cc:260] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:24.972975 13623 raft_consensus.cc:348] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:24.973702 13623 raft_consensus.cc:374] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:24.973981 13623 raft_consensus.cc:729] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:24.974354 13622 tablet_bootstrap.cc:492] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:24.974908 13622 ts_tablet_manager.cc:1397] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.157s user 0.007s sys 0.010s
I20260806 20:33:24.974666 13623 consensus_queue.cc:260] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:24.979192 13604 heartbeater.cc:503] Master 127.12.248.190:33869 was elected leader, sending a full tablet report...
I20260806 20:33:24.979619 13624 ts_tablet_manager.cc:1428] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.013s user 0.003s sys 0.010s
I20260806 20:33:24.980654 13624 tablet_bootstrap.cc:492] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:24.981685 13623 ts_tablet_manager.cc:1428] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260806 20:33:24.982338 13528 heartbeater.cc:503] Master 127.12.248.190:33869 was elected leader, sending a full tablet report...
W20260806 20:33:24.982597 13605 tablet.cc:2369] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:33:24.978446 13622 raft_consensus.cc:348] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:24.986944 13624 tablet_bootstrap.cc:654] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:24.986877 13622 raft_consensus.cc:374] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:24.987409 13622 raft_consensus.cc:729] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:24.988061 13623 tablet_bootstrap.cc:492] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:24.988250 13622 consensus_queue.cc:260] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:24.994396 13629 raft_consensus.cc:482] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:24.994905 13629 raft_consensus.cc:504] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.000388 13624 tablet_bootstrap.cc:492] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.001048 13624 ts_tablet_manager.cc:1397] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20260806 20:33:25.004861 13624 raft_consensus.cc:348] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.005908 13624 raft_consensus.cc:374] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.006265 13624 raft_consensus.cc:729] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.007777 13623 tablet_bootstrap.cc:654] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.007560 13629 leader_election.cc:288] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.014663 13624 consensus_queue.cc:260] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.029699 13622 ts_tablet_manager.cc:1428] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.054s user 0.024s sys 0.014s
I20260806 20:33:25.030720 13622 tablet_bootstrap.cc:492] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:25.035684 13624 ts_tablet_manager.cc:1428] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.034s user 0.010s sys 0.000s
I20260806 20:33:25.038127 13624 tablet_bootstrap.cc:492] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:25.043430 13622 tablet_bootstrap.cc:654] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.044245 13623 tablet_bootstrap.cc:492] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.044811 13623 ts_tablet_manager.cc:1397] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.008s
I20260806 20:33:25.047573 13623 raft_consensus.cc:348] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.048345 13623 raft_consensus.cc:374] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.048686 13623 raft_consensus.cc:729] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.050587 13623 consensus_queue.cc:260] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.057011 13624 tablet_bootstrap.cc:654] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.063593 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.065002 13623 ts_tablet_manager.cc:1428] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
W20260806 20:33:25.067947 13464 leader_election.cc:341] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.071373 13623 tablet_bootstrap.cc:492] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:25.071542 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
W20260806 20:33:25.074028 13462 leader_election.cc:341] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.074470 13462 leader_election.cc:302] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9819c221b89b4b3c829980ffc65b3679; no voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73
I20260806 20:33:25.075349 13629 raft_consensus.cc:2726] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:25.079854 13623 tablet_bootstrap.cc:654] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.083943 13624 tablet_bootstrap.cc:492] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.084586 13624 ts_tablet_manager.cc:1397] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.048s user 0.021s sys 0.002s
I20260806 20:33:25.088119 13622 tablet_bootstrap.cc:492] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.088682 13622 ts_tablet_manager.cc:1397] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.058s user 0.017s sys 0.034s
I20260806 20:33:25.088136 13624 raft_consensus.cc:348] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.089633 13624 raft_consensus.cc:374] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.090147 13624 raft_consensus.cc:729] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.091260 13624 consensus_queue.cc:260] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.091377 13622 raft_consensus.cc:348] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.092657 13622 raft_consensus.cc:374] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.092963 13622 raft_consensus.cc:729] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.093675 13622 consensus_queue.cc:260] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.094671 13624 ts_tablet_manager.cc:1428] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260806 20:33:25.095043 13623 tablet_bootstrap.cc:492] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.095499 13623 ts_tablet_manager.cc:1397] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.002s
I20260806 20:33:25.095803 13622 ts_tablet_manager.cc:1428] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 20:33:25.096719 13622 tablet_bootstrap.cc:492] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:25.098222 13623 raft_consensus.cc:348] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.098976 13623 raft_consensus.cc:374] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.099255 13623 raft_consensus.cc:729] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.099921 13623 consensus_queue.cc:260] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.102581 13623 ts_tablet_manager.cc:1428] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260806 20:33:25.102871 13622 tablet_bootstrap.cc:654] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.103619 13623 tablet_bootstrap.cc:492] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:25.110373 13623 tablet_bootstrap.cc:654] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.122419 13623 tablet_bootstrap.cc:492] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.122892 13623 ts_tablet_manager.cc:1397] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260806 20:33:25.125634 13623 raft_consensus.cc:348] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.126379 13623 raft_consensus.cc:374] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.126667 13623 raft_consensus.cc:729] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.126927 13624 tablet_bootstrap.cc:492] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:25.127344 13623 consensus_queue.cc:260] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.130496 13623 ts_tablet_manager.cc:1428] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 20:33:25.131374 13623 tablet_bootstrap.cc:492] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:25.137620 13623 tablet_bootstrap.cc:654] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.148000 13624 tablet_bootstrap.cc:654] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.149670 13623 tablet_bootstrap.cc:492] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.150182 13623 ts_tablet_manager.cc:1397] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260806 20:33:25.153119 13623 raft_consensus.cc:348] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.153839 13623 raft_consensus.cc:374] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.153920 13622 tablet_bootstrap.cc:492] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.154636 13622 ts_tablet_manager.cc:1397] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.058s user 0.030s sys 0.019s
I20260806 20:33:25.154135 13623 raft_consensus.cc:729] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.155524 13623 consensus_queue.cc:260] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.157740 13623 ts_tablet_manager.cc:1428] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:33:25.158138 13622 raft_consensus.cc:348] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.158774 13623 tablet_bootstrap.cc:492] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:25.158860 13622 raft_consensus.cc:374] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.159219 13622 raft_consensus.cc:729] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.159987 13622 consensus_queue.cc:260] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.162411 13622 ts_tablet_manager.cc:1428] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:33:25.163290 13622 tablet_bootstrap.cc:492] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:25.165349 13623 tablet_bootstrap.cc:654] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.170243 13622 tablet_bootstrap.cc:654] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.177086 13623 tablet_bootstrap.cc:492] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.177538 13623 ts_tablet_manager.cc:1397] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20260806 20:33:25.180119 13624 tablet_bootstrap.cc:492] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.180019 13623 raft_consensus.cc:348] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.180799 13623 raft_consensus.cc:374] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.180963 13624 ts_tablet_manager.cc:1397] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.007s
I20260806 20:33:25.181092 13623 raft_consensus.cc:729] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.182027 13623 consensus_queue.cc:260] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.184443 13623 ts_tablet_manager.cc:1428] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:33:25.185410 13623 tablet_bootstrap.cc:492] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:25.191501 13623 tablet_bootstrap.cc:654] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.192915 13622 tablet_bootstrap.cc:492] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.193368 13622 ts_tablet_manager.cc:1397] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.003s
I20260806 20:33:25.196126 13622 raft_consensus.cc:348] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.196888 13622 raft_consensus.cc:374] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.197166 13622 raft_consensus.cc:729] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.197911 13622 consensus_queue.cc:260] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.200387 13622 ts_tablet_manager.cc:1428] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:33:25.200923 13628 raft_consensus.cc:482] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.201390 13628 raft_consensus.cc:504] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.211125 13623 tablet_bootstrap.cc:492] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.211578 13623 ts_tablet_manager.cc:1397] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.018s
I20260806 20:33:25.214078 13623 raft_consensus.cc:348] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.214397 13628 leader_election.cc:288] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.214736 13623 raft_consensus.cc:374] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.215041 13623 raft_consensus.cc:729] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.215709 13623 consensus_queue.cc:260] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.216001 13624 raft_consensus.cc:348] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.219197 13623 ts_tablet_manager.cc:1428] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260806 20:33:25.220209 13623 tablet_bootstrap.cc:492] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:25.201375 13622 tablet_bootstrap.cc:492] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:25.226838 13623 tablet_bootstrap.cc:654] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.217141 13624 raft_consensus.cc:374] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.233359 13624 raft_consensus.cc:729] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.234321 13624 consensus_queue.cc:260] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.237926 13624 ts_tablet_manager.cc:1428] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.056s user 0.006s sys 0.001s
I20260806 20:33:25.238662 13623 tablet_bootstrap.cc:492] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.239424 13624 tablet_bootstrap.cc:492] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:25.239252 13623 ts_tablet_manager.cc:1397] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260806 20:33:25.240319 13622 tablet_bootstrap.cc:654] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.247560 13623 raft_consensus.cc:348] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.248242 13623 raft_consensus.cc:374] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.248507 13623 raft_consensus.cc:729] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.251255 13623 consensus_queue.cc:260] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.253731 13629 raft_consensus.cc:482] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.254160 13629 raft_consensus.cc:504] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.259843 13629 leader_election.cc:288] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.253547 13623 ts_tablet_manager.cc:1428] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 20:33:25.261044 13623 tablet_bootstrap.cc:492] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: Bootstrap starting.
I20260806 20:33:25.261435 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.262759 13422 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 0.
I20260806 20:33:25.264580 13629 raft_consensus.cc:482] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.265031 13629 raft_consensus.cc:504] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.265327 13464 leader_election.cc:302] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.266829 13629 leader_election.cc:288] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.267287 13629 raft_consensus.cc:2781] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.267378 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
I20260806 20:33:25.267618 13629 raft_consensus.cc:482] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.267972 13629 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.268047 13579 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 0.
I20260806 20:33:25.270908 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
W20260806 20:33:25.272782 13462 leader_election.cc:341] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.274212 13624 tablet_bootstrap.cc:654] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.276015 13623 tablet_bootstrap.cc:654] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.283854 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.284802 13422 raft_consensus.cc:2454] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 0.
I20260806 20:33:25.285733 13629 raft_consensus.cc:504] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.286681 13464 leader_election.cc:302] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters: 038a7b6ab86f465884b0861c91d19d36
I20260806 20:33:25.287706 13629 leader_election.cc:288] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.289393 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.289968 13422 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.288333 13629 raft_consensus.cc:2781] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.290536 13629 raft_consensus.cc:482] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.290891 13629 raft_consensus.cc:3037] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.291528 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:25.292372 13579 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.297049 13422 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 1.
I20260806 20:33:25.298255 13464 leader_election.cc:302] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.298808 13579 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 1.
I20260806 20:33:25.313068 13653 raft_consensus.cc:482] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.313205 13629 raft_consensus.cc:504] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.313592 13653 raft_consensus.cc:504] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.317096 13628 raft_consensus.cc:482] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.318199 13629 leader_election.cc:288] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.319118 13622 tablet_bootstrap.cc:492] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.320097 13622 ts_tablet_manager.cc:1397] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.119s user 0.017s sys 0.003s
I20260806 20:33:25.319528 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42329950ce5e4e14baf56b8caa9710f1" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.317561 13628 raft_consensus.cc:504] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.319142 13629 raft_consensus.cc:2781] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:33:25.339831 13542 leader_election.cc:341] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.341645 13628 leader_election.cc:288] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.338804 13630 raft_consensus.cc:482] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.332571 13653 leader_election.cc:288] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.343641 13629 raft_consensus.cc:686] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:25.347882 13630 raft_consensus.cc:504] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.346325 13622 raft_consensus.cc:348] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.350780 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.350940 13622 raft_consensus.cc:374] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.348403 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:25.348755 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
W20260806 20:33:25.352062 13542 leader_election.cc:341] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.352240 13622 raft_consensus.cc:729] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.354753 13624 tablet_bootstrap.cc:492] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.354562 13502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42329950ce5e4e14baf56b8caa9710f1" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
W20260806 20:33:25.351622 13464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:45137, user_credentials={real_user=slave}} blocked reactor thread for 48940.1us
W20260806 20:33:25.355463 13462 leader_election.cc:341] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.356721 13502 raft_consensus.cc:2454] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 0.
I20260806 20:33:25.355554 13577 raft_consensus.cc:2454] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 0.
I20260806 20:33:25.357569 13630 leader_election.cc:288] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.355958 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.358047 13624 ts_tablet_manager.cc:1397] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.119s user 0.016s sys 0.011s
I20260806 20:33:25.355480 13629 consensus_queue.cc:237] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.359519 13623 tablet_bootstrap.cc:492] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: No bootstrap required, opened a new log
I20260806 20:33:25.360895 13623 ts_tablet_manager.cc:1397] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: Time spent bootstrapping tablet: real 0.100s user 0.024s sys 0.028s
I20260806 20:33:25.365166 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.365893 13422 raft_consensus.cc:3037] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.367229 13462 leader_election.cc:302] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.367403 13624 raft_consensus.cc:348] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.368001 13653 raft_consensus.cc:2781] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.368170 13624 raft_consensus.cc:374] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.368552 13624 raft_consensus.cc:729] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.368389 13653 raft_consensus.cc:482] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.368984 13653 raft_consensus.cc:3037] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:33:25.372865 13540 leader_election.cc:341] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.373219 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.373468 13540 leader_election.cc:302] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36; no voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:25.371251 13624 consensus_queue.cc:260] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.374652 13421 raft_consensus.cc:2454] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 0.
I20260806 20:33:25.374940 13628 raft_consensus.cc:2726] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:25.375727 13540 leader_election.cc:302] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters: 6430856814214be2be901e26e0c9ae73
I20260806 20:33:25.376539 13628 raft_consensus.cc:2781] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.376891 13628 raft_consensus.cc:482] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.377205 13628 raft_consensus.cc:3037] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.359105 13622 consensus_queue.cc:260] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.379033 13623 raft_consensus.cc:348] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.379799 13623 raft_consensus.cc:374] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.380101 13623 raft_consensus.cc:729] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.382694 13622 ts_tablet_manager.cc:1428] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.062s user 0.008s sys 0.000s
I20260806 20:33:25.383694 13622 tablet_bootstrap.cc:492] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:25.385769 13623 consensus_queue.cc:260] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.389678 13422 raft_consensus.cc:2454] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 1.
I20260806 20:33:25.390619 13623 ts_tablet_manager.cc:1428] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20260806 20:33:25.391131 13464 leader_election.cc:302] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters: 038a7b6ab86f465884b0861c91d19d36
I20260806 20:33:25.392283 13629 raft_consensus.cc:2781] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:25.393064 13629 raft_consensus.cc:686] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:25.394480 13629 consensus_queue.cc:237] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.396225 13653 raft_consensus.cc:504] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.403306 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.403887 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:25.404568 13577 raft_consensus.cc:3037] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.401803 13653 leader_election.cc:288] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.405233 13622 tablet_bootstrap.cc:654] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.407159 13628 raft_consensus.cc:504] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.406833 13422 raft_consensus.cc:3037] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.413393 13624 ts_tablet_manager.cc:1428] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.054s user 0.012s sys 0.003s
I20260806 20:33:25.413516 13628 leader_election.cc:288] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.414376 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42329950ce5e4e14baf56b8caa9710f1" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.414633 13624 tablet_bootstrap.cc:492] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:25.415002 13421 raft_consensus.cc:3037] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.425064 13502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42329950ce5e4e14baf56b8caa9710f1" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:25.427034 13502 raft_consensus.cc:3037] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.427654 13577 raft_consensus.cc:2454] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 1.
I20260806 20:33:25.438432 13462 leader_election.cc:302] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.439850 13653 raft_consensus.cc:2781] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:25.440994 13653 raft_consensus.cc:686] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:25.443377 13653 consensus_queue.cc:237] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.451848 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.453694 13501 raft_consensus.cc:2454] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 0.
I20260806 20:33:25.456837 13628 raft_consensus.cc:482] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.461225 13624 tablet_bootstrap.cc:654] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.460546 13628 raft_consensus.cc:504] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.464514 13383 leader_election.cc:302] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.470356 13630 raft_consensus.cc:2781] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.472504 13630 raft_consensus.cc:482] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.472760 13502 raft_consensus.cc:2454] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 1.
I20260806 20:33:25.473346 13630 raft_consensus.cc:3037] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.475965 13422 raft_consensus.cc:2454] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 1.
I20260806 20:33:25.476238 13540 leader_election.cc:302] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.475360 13421 raft_consensus.cc:2454] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 1.
I20260806 20:33:25.475749 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
I20260806 20:33:25.478264 13659 raft_consensus.cc:2781] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:33:25.480137 13383 leader_election.cc:341] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.484827 13628 leader_election.cc:288] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.485978 13622 tablet_bootstrap.cc:492] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.491492 13502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.493645 13659 raft_consensus.cc:686] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:25.495453 13502 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 0.
I20260806 20:33:25.502192 13659 consensus_queue.cc:237] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.501339 13540 leader_election.cc:302] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.505512 13628 raft_consensus.cc:2781] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.505505 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.506109 13628 raft_consensus.cc:482] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.506472 13628 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:33:25.507117 13542 leader_election.cc:341] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.508596 13630 raft_consensus.cc:504] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.512746 13624 tablet_bootstrap.cc:492] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.513334 13624 ts_tablet_manager.cc:1397] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.099s user 0.007s sys 0.010s
I20260806 20:33:25.513934 13628 raft_consensus.cc:504] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.517321 13624 raft_consensus.cc:348] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.518471 13624 raft_consensus.cc:374] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.518847 13624 raft_consensus.cc:729] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.505290 13622 ts_tablet_manager.cc:1397] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.122s user 0.021s sys 0.016s
I20260806 20:33:25.519733 13624 consensus_queue.cc:260] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.523197 13653 raft_consensus.cc:482] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.524359 13502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:25.525030 13502 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.524940 13653 raft_consensus.cc:504] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.527197 13653 leader_election.cc:288] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.529044 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
I20260806 20:33:25.530422 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
W20260806 20:33:25.530977 13462 leader_election.cc:341] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.521552 13628 leader_election.cc:288] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.533406 13630 leader_election.cc:288] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 election: Requested vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
W20260806 20:33:25.532612 13464 leader_election.cc:341] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.534557 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.535226 13464 leader_election.cc:302] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9819c221b89b4b3c829980ffc65b3679; no voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73
I20260806 20:33:25.535264 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:25.535246 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:25.544297 13653 raft_consensus.cc:2726] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:25.533047 13314 catalog_manager.cc:5495] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 1 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:25.543293 13624 ts_tablet_manager.cc:1428] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.030s user 0.010s sys 0.001s
W20260806 20:33:25.543335 13542 leader_election.cc:341] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.544045 13501 raft_consensus.cc:3037] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:33:25.545660 13383 leader_election.cc:341] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:25.544035 13622 raft_consensus.cc:348] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.546058 13624 tablet_bootstrap.cc:492] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:25.540985 13502 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 1.
I20260806 20:33:25.546820 13622 raft_consensus.cc:374] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.548264 13622 raft_consensus.cc:729] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.548834 13540 leader_election.cc:302] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters: 6430856814214be2be901e26e0c9ae73
I20260806 20:33:25.550958 13628 raft_consensus.cc:2781] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:25.551561 13628 raft_consensus.cc:686] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:25.550351 13622 consensus_queue.cc:260] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.555598 13622 ts_tablet_manager.cc:1428] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.034s user 0.007s sys 0.001s
I20260806 20:33:25.555047 13628 consensus_queue.cc:237] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.558736 13622 tablet_bootstrap.cc:492] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:25.566669 13624 tablet_bootstrap.cc:654] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.565606 13501 raft_consensus.cc:2454] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 1.
I20260806 20:33:25.569268 13383 leader_election.cc:302] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters: 038a7b6ab86f465884b0861c91d19d36
I20260806 20:33:25.571717 13630 raft_consensus.cc:2781] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:25.579198 13630 raft_consensus.cc:686] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 LEADER]: Becoming Leader. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:25.581887 13630 consensus_queue.cc:237] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.589360 13622 tablet_bootstrap.cc:654] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.599910 13624 tablet_bootstrap.cc:492] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.600797 13624 ts_tablet_manager.cc:1397] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.020s
I20260806 20:33:25.607151 13622 tablet_bootstrap.cc:492] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.607523 13622 ts_tablet_manager.cc:1397] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.016s
I20260806 20:33:25.607240 13624 raft_consensus.cc:348] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.607968 13624 raft_consensus.cc:374] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.608228 13624 raft_consensus.cc:729] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.608867 13624 consensus_queue.cc:260] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.609632 13622 raft_consensus.cc:348] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.610301 13622 raft_consensus.cc:374] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.610538 13622 raft_consensus.cc:729] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.610756 13624 ts_tablet_manager.cc:1428] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 20:33:25.611568 13622 consensus_queue.cc:260] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.611588 13624 tablet_bootstrap.cc:492] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:25.614102 13622 ts_tablet_manager.cc:1428] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 20:33:25.615006 13622 tablet_bootstrap.cc:492] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
W20260806 20:33:25.615303 13364 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:33:25.620623 13624 tablet_bootstrap.cc:654] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.620913 13622 tablet_bootstrap.cc:654] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.623524 13313 catalog_manager.cc:5495] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:25.628856 13628 raft_consensus.cc:482] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.635267 13628 raft_consensus.cc:504] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:25.637398 13628 leader_election.cc:288] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137)
I20260806 20:33:25.632627 13624 tablet_bootstrap.cc:492] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.638460 13624 ts_tablet_manager.cc:1397] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.010s
I20260806 20:33:25.638701 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.639681 13501 raft_consensus.cc:2379] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038a7b6ab86f465884b0861c91d19d36 in current term 1: Already voted for candidate 6430856814214be2be901e26e0c9ae73 in this term.
I20260806 20:33:25.640403 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.641958 13542 leader_election.cc:302] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36; no voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:25.642643 13624 raft_consensus.cc:348] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.643357 13624 raft_consensus.cc:374] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.643666 13624 raft_consensus.cc:729] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.644490 13622 tablet_bootstrap.cc:492] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.644317 13624 consensus_queue.cc:260] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.644949 13622 ts_tablet_manager.cc:1397] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.011s
I20260806 20:33:25.645689 13664 raft_consensus.cc:3037] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.646617 13624 ts_tablet_manager.cc:1428] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:33:25.647469 13624 tablet_bootstrap.cc:492] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36: Bootstrap starting.
I20260806 20:33:25.647667 13622 raft_consensus.cc:348] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.648353 13622 raft_consensus.cc:374] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.648650 13622 raft_consensus.cc:729] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.649266 13622 consensus_queue.cc:260] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.649499 13315 catalog_manager.cc:5495] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6430856814214be2be901e26e0c9ae73 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "6430856814214be2be901e26e0c9ae73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:25.651612 13622 ts_tablet_manager.cc:1428] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:33:25.652498 13622 tablet_bootstrap.cc:492] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73: Bootstrap starting.
I20260806 20:33:25.652415 13664 raft_consensus.cc:2726] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:25.654059 13624 tablet_bootstrap.cc:654] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.657626 13622 tablet_bootstrap.cc:654] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:25.689328 13622 tablet_bootstrap.cc:492] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73: No bootstrap required, opened a new log
I20260806 20:33:25.689915 13622 ts_tablet_manager.cc:1397] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.007s
I20260806 20:33:25.684298 13314 catalog_manager.cc:5495] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 1 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:25.692502 13622 raft_consensus.cc:348] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.693184 13622 raft_consensus.cc:374] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.693440 13622 raft_consensus.cc:729] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.694000 13622 consensus_queue.cc:260] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.694211 13314 catalog_manager.cc:5495] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 1 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:25.696117 13622 ts_tablet_manager.cc:1428] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:33:25.687237 13313 catalog_manager.cc:5495] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:25.704780 13664 raft_consensus.cc:482] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.705281 13664 raft_consensus.cc:504] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.707376 13664 leader_election.cc:288] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.707877 13624 tablet_bootstrap.cc:492] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36: No bootstrap required, opened a new log
I20260806 20:33:25.708289 13624 ts_tablet_manager.cc:1397] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36: Time spent bootstrapping tablet: real 0.061s user 0.023s sys 0.034s
I20260806 20:33:25.708602 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.709218 13501 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 0.
I20260806 20:33:25.710884 13624 raft_consensus.cc:348] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.711175 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.711571 13624 raft_consensus.cc:374] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:25.711372 13540 leader_election.cc:302] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.711743 13421 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 0.
I20260806 20:33:25.711866 13624 raft_consensus.cc:729] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Initialized, Role: FOLLOWER
I20260806 20:33:25.712283 13664 raft_consensus.cc:2781] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.712661 13664 raft_consensus.cc:482] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.712512 13624 consensus_queue.cc:260] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.713009 13664 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.714808 13624 ts_tablet_manager.cc:1428] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 20:33:25.718959 13664 raft_consensus.cc:504] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.720783 13664 leader_election.cc:288] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.723419 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.724076 13421 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.724102 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:25.724803 13501 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.729547 13421 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 1.
I20260806 20:33:25.730500 13542 leader_election.cc:302] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:25.731318 13664 raft_consensus.cc:2781] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:25.731801 13664 raft_consensus.cc:686] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:25.731644 13501 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 1.
I20260806 20:33:25.732609 13664 consensus_queue.cc:237] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.740125 13630 raft_consensus.cc:482] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.740599 13630 raft_consensus.cc:504] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.742493 13630 leader_election.cc:288] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.741783 13315 catalog_manager.cc:5495] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:25.743629 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.744374 13501 raft_consensus.cc:2379] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6430856814214be2be901e26e0c9ae73 in current term 1: Already voted for candidate 038a7b6ab86f465884b0861c91d19d36 in this term.
I20260806 20:33:25.745325 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
I20260806 20:33:25.746798 13383 leader_election.cc:302] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6430856814214be2be901e26e0c9ae73; no voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:25.747591 13630 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.753927 13630 raft_consensus.cc:2726] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:25.756536 13653 raft_consensus.cc:482] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.757031 13653 raft_consensus.cc:504] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.759055 13653 leader_election.cc:288] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.759793 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.760098 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36" is_pre_election: true
I20260806 20:33:25.760507 13421 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 0.
I20260806 20:33:25.760748 13577 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 0.
I20260806 20:33:25.761505 13464 leader_election.cc:302] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.762307 13653 raft_consensus.cc:2781] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.762651 13653 raft_consensus.cc:482] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.762933 13653 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.767012 13653 raft_consensus.cc:504] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.768586 13653 leader_election.cc:288] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:25.769315 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.769609 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:25.769870 13664 raft_consensus.cc:482] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.770139 13577 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.770303 13664 raft_consensus.cc:504] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.772181 13664 leader_election.cc:288] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.772713 13421 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.774478 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.775249 13422 raft_consensus.cc:2379] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038a7b6ab86f465884b0861c91d19d36 in current term 1: Already voted for candidate 9819c221b89b4b3c829980ffc65b3679 in this term.
I20260806 20:33:25.777060 13577 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 1.
I20260806 20:33:25.777243 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.778240 13462 leader_election.cc:302] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.779191 13653 raft_consensus.cc:2781] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:25.779699 13653 raft_consensus.cc:686] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:25.779615 13540 leader_election.cc:302] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36; no voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:25.780534 13421 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 1.
I20260806 20:33:25.781013 13664 raft_consensus.cc:3037] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.780660 13653 consensus_queue.cc:237] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.787297 13664 raft_consensus.cc:2726] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:25.789804 13315 catalog_manager.cc:5495] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 1 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:25.792137 13653 consensus_queue.cc:1035] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:25.807195 13653 consensus_queue.cc:1035] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:25.820299 13664 raft_consensus.cc:482] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:25.820811 13664 raft_consensus.cc:504] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.822321 13664 leader_election.cc:288] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.823048 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73" is_pre_election: true
I20260806 20:33:25.823361 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679" is_pre_election: true
I20260806 20:33:25.823644 13421 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 0.
I20260806 20:33:25.824007 13501 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 0.
I20260806 20:33:25.825220 13540 leader_election.cc:302] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:25.825982 13664 raft_consensus.cc:2781] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:25.826324 13664 raft_consensus.cc:482] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:25.826661 13664 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.832991 13664 raft_consensus.cc:504] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.834857 13664 leader_election.cc:288] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:25.835572 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:25.835979 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:25.836143 13421 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.836591 13501 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:25.842865 13421 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 1.
I20260806 20:33:25.842870 13501 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 1.
I20260806 20:33:25.844206 13542 leader_election.cc:302] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:25.845072 13664 raft_consensus.cc:2781] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:25.845546 13664 raft_consensus.cc:686] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:25.846294 13664 consensus_queue.cc:237] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:25.855603 13313 catalog_manager.cc:5495] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:25.879495 13653 consensus_queue.cc:1035] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:25.894686 13629 consensus_queue.cc:1035] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:25.912390 13629 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:25.923195 13656 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:25.928763 13628 consensus_queue.cc:1035] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:25.946063 13664 consensus_queue.cc:1035] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:26.031826 13664 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.036084 13500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:59752
I20260806 20:33:26.036939 13500 raft_consensus.cc:595] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Received request to transfer leadership to TS 6430856814214be2be901e26e0c9ae73
I20260806 20:33:26.037545 13630 consensus_queue.cc:1035] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.057371 13628 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.058275 13577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
mode: GRACEFUL
new_leader_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:48026
I20260806 20:33:26.058970 13577 raft_consensus.cc:595] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Received request to transfer leadership to TS 6430856814214be2be901e26e0c9ae73
I20260806 20:33:26.064812 13502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:59752
I20260806 20:33:26.065348 13502 raft_consensus.cc:595] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Received request to transfer leadership to TS 6430856814214be2be901e26e0c9ae73
I20260806 20:33:26.070626 13630 consensus_queue.cc:1035] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.074267 13579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
mode: GRACEFUL
new_leader_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:48026
I20260806 20:33:26.074848 13579 raft_consensus.cc:595] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Received request to transfer leadership to TS 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:26.087245 13579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
mode: GRACEFUL
new_leader_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:48026
I20260806 20:33:26.087831 13579 raft_consensus.cc:595] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Received request to transfer leadership to TS 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:26.115861 13628 raft_consensus.cc:979] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36: : Instructing follower 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:26.116554 13664 raft_consensus.cc:1067] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Signalling peer 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:26.118803 13421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:40534
I20260806 20:33:26.121711 13421 raft_consensus.cc:482] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:26.122170 13421 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.128811 13421 raft_consensus.cc:504] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.130765 13421 leader_election.cc:288] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Requested vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:26.131819 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:26.131819 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:26.132659 13500 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.132694 13579 raft_consensus.cc:3032] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:26.133234 13579 raft_consensus.cc:729] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:26.134083 13579 consensus_queue.cc:260] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.135474 13579 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.139676 13500 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:26.140784 13383 leader_election.cc:302] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:26.141769 13703 raft_consensus.cc:2781] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:26.141963 13579 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:26.142302 13703 raft_consensus.cc:686] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Becoming Leader. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:26.143157 13703 consensus_queue.cc:237] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.150280 13314 catalog_manager.cc:5495] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 reported cstate change: term changed from 1 to 2, leader changed from 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131) to 6430856814214be2be901e26e0c9ae73 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "6430856814214be2be901e26e0c9ae73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:26.168018 13664 consensus_queue.cc:1035] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.188853 13705 consensus_queue.cc:1035] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:26.196362 13659 raft_consensus.cc:979] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36: : Instructing follower 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:26.196864 13705 raft_consensus.cc:1067] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Signalling peer 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:26.198375 13500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:59728
I20260806 20:33:26.198879 13500 raft_consensus.cc:482] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:26.199193 13500 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.204265 13500 raft_consensus.cc:504] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.206107 13500 leader_election.cc:288] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 2 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:26.207062 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:26.207675 13421 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.213559 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:26.214052 13579 raft_consensus.cc:3032] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:26.214327 13579 raft_consensus.cc:729] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:26.214216 13421 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 2.
I20260806 20:33:26.215457 13464 leader_election.cc:302] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:26.215178 13579 consensus_queue.cc:260] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.216321 13653 raft_consensus.cc:2781] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:26.216593 13579 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.216854 13653 raft_consensus.cc:686] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:26.217701 13653 consensus_queue.cc:237] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.222388 13579 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 2.
I20260806 20:33:26.225075 13313 catalog_manager.cc:5495] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 1 to 2, leader changed from 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131) to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 2 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:26.272634 13629 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.293200 13629 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.302140 13653 raft_consensus.cc:979] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:26.302639 13629 raft_consensus.cc:1067] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Signalling peer 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:26.304229 13421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:40526
I20260806 20:33:26.304831 13421 raft_consensus.cc:482] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:26.305210 13421 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.309819 13421 raft_consensus.cc:504] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.311564 13421 leader_election.cc:288] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Requested vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:26.312294 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:26.312637 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:26.312918 13500 raft_consensus.cc:3032] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:26.313210 13579 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.313278 13500 raft_consensus.cc:729] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:26.314146 13500 consensus_queue.cc:260] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.315358 13500 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.319723 13579 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:26.320681 13383 leader_election.cc:302] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:26.321362 13703 raft_consensus.cc:2781] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:26.321735 13500 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:26.321798 13703 raft_consensus.cc:686] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Becoming Leader. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:26.322765 13703 consensus_queue.cc:237] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.329016 13314 catalog_manager.cc:5495] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 reported cstate change: term changed from 1 to 2, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 6430856814214be2be901e26e0c9ae73 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "6430856814214be2be901e26e0c9ae73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:26.331840 13705 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.335450 13659 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:26.364858 13659 raft_consensus.cc:979] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: : Instructing follower 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:26.365788 13628 raft_consensus.cc:1067] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Signalling peer 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:26.369731 13500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:59728
I20260806 20:33:26.370362 13500 raft_consensus.cc:482] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:26.370817 13500 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.378413 13500 raft_consensus.cc:504] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.380677 13500 leader_election.cc:288] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 2 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:26.381129 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:26.381743 13422 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.386273 13422 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 2.
I20260806 20:33:26.386945 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:26.387307 13464 leader_election.cc:302] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:26.387584 13579 raft_consensus.cc:3032] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:26.387897 13579 raft_consensus.cc:729] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:26.388149 13629 raft_consensus.cc:2781] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:26.388662 13629 raft_consensus.cc:686] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:26.388734 13579 consensus_queue.cc:260] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.389556 13629 consensus_queue.cc:237] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.389999 13579 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.396052 13579 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 2.
I20260806 20:33:26.397763 13314 catalog_manager.cc:5495] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 1 to 2, leader changed from 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131) to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 2 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:26.439891 13629 raft_consensus.cc:979] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:26.440341 13656 raft_consensus.cc:1067] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Signalling peer 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:26.441941 13422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:40526
I20260806 20:33:26.442425 13422 raft_consensus.cc:482] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:26.442739 13422 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.447392 13422 raft_consensus.cc:504] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.448984 13422 leader_election.cc:288] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Requested vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:26.449709 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:26.450170 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:26.450299 13500 raft_consensus.cc:3032] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:26.450766 13577 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.450733 13500 raft_consensus.cc:729] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:26.451749 13500 consensus_queue.cc:260] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.453042 13500 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:26.457581 13577 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:26.458709 13383 leader_election.cc:302] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:26.459498 13703 raft_consensus.cc:2781] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:26.459555 13500 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:26.460116 13703 raft_consensus.cc:686] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Becoming Leader. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:26.461004 13703 consensus_queue.cc:237] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:26.468303 13314 catalog_manager.cc:5495] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 reported cstate change: term changed from 1 to 2, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 6430856814214be2be901e26e0c9ae73 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "6430856814214be2be901e26e0c9ae73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:26.552609 13500 raft_consensus.cc:1261] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.553776 13703 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:26.567997 13577 raft_consensus.cc:1261] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.570399 13703 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:26.692154 13577 raft_consensus.cc:1261] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.693667 13656 consensus_queue.cc:1035] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:26.701261 13422 raft_consensus.cc:1261] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.702806 13653 consensus_queue.cc:1035] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:26.804106 13500 raft_consensus.cc:1261] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.805388 13703 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:26.814162 13577 raft_consensus.cc:1261] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.815559 13669 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:26.829612 13422 raft_consensus.cc:1261] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.830773 13653 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:26.837472 13577 raft_consensus.cc:1261] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:26.838856 13629 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:27.036583 13577 raft_consensus.cc:1261] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:27.037778 13669 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:27.045604 13500 raft_consensus.cc:1261] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:27.047199 13669 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:29.141764 13282 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260806 20:33:29.142671 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:29.167232 13500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:38946
I20260806 20:33:29.167793 13500 raft_consensus.cc:595] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Received request to transfer leadership to TS 038a7b6ab86f465884b0861c91d19d36
I20260806 20:33:29.180325 13422 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
mode: GRACEFUL
new_leader_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:44190
I20260806 20:33:29.180891 13422 raft_consensus.cc:595] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Received request to transfer leadership to TS 038a7b6ab86f465884b0861c91d19d36
I20260806 20:33:29.181841 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 20:33:29.182301 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:29.426391 13629 raft_consensus.cc:979] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:29.426877 13718 raft_consensus.cc:1067] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Signalling peer 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:29.428373 13579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:48008
I20260806 20:33:29.428887 13579 raft_consensus.cc:482] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:29.429211 13579 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:29.433827 13579 raft_consensus.cc:504] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:29.435432 13579 leader_election.cc:288] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 3 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:29.436269 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:29.436508 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:29.436864 13422 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:29.437057 13500 raft_consensus.cc:3032] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:33:29.437398 13500 raft_consensus.cc:729] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:29.438041 13500 consensus_queue.cc:260] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:29.438933 13500 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:29.441308 13422 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 3.
I20260806 20:33:29.442381 13542 leader_election.cc:302] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:29.443208 13659 raft_consensus.cc:2781] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:33:29.443632 13500 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 3.
I20260806 20:33:29.444423 13659 raft_consensus.cc:686] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:29.445240 13659 consensus_queue.cc:237] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:29.453296 13313 catalog_manager.cc:5495] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 2 to 3, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 3 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:29.511410 13714 raft_consensus.cc:979] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73: : Instructing follower 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:29.511852 13726 raft_consensus.cc:1067] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Signalling peer 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:29.513219 13579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:48010
I20260806 20:33:29.513715 13579 raft_consensus.cc:482] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:29.514011 13579 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:29.518378 13579 raft_consensus.cc:504] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:29.519932 13579 leader_election.cc:288] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 3 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:29.520741 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:29.520990 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:29.521278 13422 raft_consensus.cc:3032] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:33:29.521572 13500 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:29.521560 13422 raft_consensus.cc:729] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:29.522360 13422 consensus_queue.cc:260] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:29.523244 13422 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:29.526175 13500 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 3.
I20260806 20:33:29.527091 13540 leader_election.cc:302] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:29.527561 13422 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 3.
I20260806 20:33:29.527911 13659 raft_consensus.cc:2781] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:33:29.528429 13659 raft_consensus.cc:686] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:29.529263 13659 consensus_queue.cc:237] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:29.535456 13313 catalog_manager.cc:5495] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 2 to 3, leader changed from 6430856814214be2be901e26e0c9ae73 (127.12.248.129) to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 3 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:29.865948 13422 raft_consensus.cc:1261] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:29.867071 13659 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:29.875836 13500 raft_consensus.cc:1261] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:29.878212 13659 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:29.934820 13422 raft_consensus.cc:1261] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:29.936048 13659 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:29.947055 13500 raft_consensus.cc:1261] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:29.948715 13719 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:30.183188 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:30.188076 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:30.188577 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:31.189486 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:31.194437 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:31.194801 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:32.195981 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:32.202620 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:32.203092 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:33.203956 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:33.209163 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:33.209525 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:34.210263 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:34.214794 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:34.215140 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:35.215853 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:35.220335 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:35.220690 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:36.221381 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:36.227566 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:36.227898 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:37.228636 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:37.233088 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:37.233407 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:38.234174 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:38.240191 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:38.240886 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:39.269449 13282 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260806 20:33:39.285595 13422 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
mode: GRACEFUL
new_leader_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:60142
I20260806 20:33:39.286234 13422 raft_consensus.cc:595] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 LEADER]: Received request to transfer leadership to TS 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:39.291010 13579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
mode: GRACEFUL
new_leader_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:48026
I20260806 20:33:39.291522 13579 raft_consensus.cc:595] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Received request to transfer leadership to TS 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:39.296129 13422 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
mode: GRACEFUL
new_leader_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:60142
I20260806 20:33:39.296586 13422 raft_consensus.cc:595] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Received request to transfer leadership to TS 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:39.300654 13422 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
mode: GRACEFUL
new_leader_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:60142
I20260806 20:33:39.301023 13422 raft_consensus.cc:595] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Received request to transfer leadership to TS 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:39.305281 13579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
mode: GRACEFUL
new_leader_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:48026
I20260806 20:33:39.305763 13579 raft_consensus.cc:595] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Received request to transfer leadership to TS 9819c221b89b4b3c829980ffc65b3679
I20260806 20:33:39.362704 13748 raft_consensus.cc:979] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73: : Instructing follower 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.363277 13743 raft_consensus.cc:1067] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Signalling peer 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.364917 13501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:59744
I20260806 20:33:39.365602 13501 raft_consensus.cc:482] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:39.366014 13501 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:39.371697 13501 raft_consensus.cc:504] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.373579 13501 leader_election.cc:288] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 3 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:39.374619 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:39.374714 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:39.375406 13422 raft_consensus.cc:3032] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:33:39.375561 13579 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:39.375759 13422 raft_consensus.cc:729] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:39.376590 13422 consensus_queue.cc:260] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.377543 13422 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:39.380195 13579 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 3.
I20260806 20:33:39.381376 13462 leader_election.cc:302] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:39.381848 13422 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 3.
I20260806 20:33:39.382246 13737 raft_consensus.cc:2781] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:33:39.383245 13737 raft_consensus.cc:686] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:39.384078 13737 consensus_queue.cc:237] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.390928 13313 catalog_manager.cc:5495] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 2 to 3, leader changed from 6430856814214be2be901e26e0c9ae73 (127.12.248.129) to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 3 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:39.448081 13748 raft_consensus.cc:979] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73: : Instructing follower 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.448578 13743 raft_consensus.cc:1067] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Signalling peer 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.449995 13501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:59744
I20260806 20:33:39.450481 13501 raft_consensus.cc:482] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:39.450770 13501 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:39.455313 13501 raft_consensus.cc:504] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.457465 13501 leader_election.cc:288] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 3 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:39.457839 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:39.458420 13422 raft_consensus.cc:3032] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:33:39.458514 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:39.458706 13422 raft_consensus.cc:729] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:39.459093 13579 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:39.459514 13422 consensus_queue.cc:260] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.460559 13422 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:39.463476 13579 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 3.
I20260806 20:33:39.464382 13462 leader_election.cc:302] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:39.465128 13737 raft_consensus.cc:2781] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:33:39.465493 13737 raft_consensus.cc:686] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:39.466399 13422 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 3.
I20260806 20:33:39.466246 13737 consensus_queue.cc:237] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.472735 13313 catalog_manager.cc:5495] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 2 to 3, leader changed from 6430856814214be2be901e26e0c9ae73 (127.12.248.129) to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 3 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:39.541608 13659 raft_consensus.cc:979] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: : Instructing follower 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.542120 13756 raft_consensus.cc:1067] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Signalling peer 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.543442 13501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:59728
I20260806 20:33:39.543923 13501 raft_consensus.cc:482] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:39.544243 13501 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:39.548251 13748 raft_consensus.cc:979] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73: : Instructing follower 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.548733 13743 raft_consensus.cc:1067] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 LEADER]: Signalling peer 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.550186 13500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:59744
I20260806 20:33:39.550107 13501 raft_consensus.cc:504] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.550642 13500 raft_consensus.cc:482] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:39.550971 13500 raft_consensus.cc:3037] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:39.551744 13501 leader_election.cc:288] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 4 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:39.552572 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:39.552850 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbd7ac041ae408bbac21f2768bf5279" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:39.553110 13422 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:39.553414 13579 raft_consensus.cc:3032] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:33:39.553704 13579 raft_consensus.cc:729] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:39.554337 13579 consensus_queue.cc:260] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.555287 13579 raft_consensus.cc:3037] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:39.556988 13500 raft_consensus.cc:504] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:39.559350 13422 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 4.
I20260806 20:33:39.559664 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:39.559976 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6eae037fbe24283b5b6bfe3a6c9c699" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:39.560240 13577 raft_consensus.cc:3037] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:39.560287 13579 raft_consensus.cc:2454] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 4.
I20260806 20:33:39.560443 13421 raft_consensus.cc:3032] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:39.560920 13421 raft_consensus.cc:729] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:39.561527 13462 leader_election.cc:302] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:39.561678 13421 consensus_queue.cc:260] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:39.562402 13737 raft_consensus.cc:2781] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:33:39.562831 13421 raft_consensus.cc:3037] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:39.562913 13737 raft_consensus.cc:686] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 4 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:39.564021 13737 consensus_queue.cc:237] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.567145 13577 raft_consensus.cc:2454] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 2.
I20260806 20:33:39.567703 13421 raft_consensus.cc:2454] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 2.
I20260806 20:33:39.568320 13462 leader_election.cc:302] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:39.569409 13749 raft_consensus.cc:2781] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:39.569797 13500 leader_election.cc:288] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 2 election: Requested vote from peers 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989), 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137)
I20260806 20:33:39.571025 13749 raft_consensus.cc:686] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:39.571933 13749 consensus_queue.cc:237] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } }
I20260806 20:33:39.573426 13313 catalog_manager.cc:5495] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 3 to 4, leader changed from 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131) to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 4 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:39.584880 13313 catalog_manager.cc:5495] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 1 to 2, leader changed from 6430856814214be2be901e26e0c9ae73 (127.12.248.129) to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 2 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:39.626932 13659 raft_consensus.cc:979] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36: : Instructing follower 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.627379 13756 raft_consensus.cc:1067] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Signalling peer 9819c221b89b4b3c829980ffc65b3679 to start an election
I20260806 20:33:39.628795 13500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
from {username='slave'} at 127.0.0.1:59728
I20260806 20:33:39.629187 13500 raft_consensus.cc:482] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:39.629398 13500 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:39.633215 13500 raft_consensus.cc:504] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.634482 13500 leader_election.cc:288] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 4 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:39.635342 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:39.635401 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a9ed1966eb4cc6968fb4e1b419e47a" candidate_uuid: "9819c221b89b4b3c829980ffc65b3679" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:39.635942 13421 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:39.636020 13577 raft_consensus.cc:3032] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:33:39.636389 13577 raft_consensus.cc:729] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:39.637141 13577 consensus_queue.cc:260] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.638083 13577 raft_consensus.cc:3037] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:39.640663 13421 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 4.
I20260806 20:33:39.641726 13464 leader_election.cc:302] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6430856814214be2be901e26e0c9ae73, 9819c221b89b4b3c829980ffc65b3679; no voters:
I20260806 20:33:39.642469 13577 raft_consensus.cc:2454] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9819c221b89b4b3c829980ffc65b3679 in term 4.
I20260806 20:33:39.642550 13749 raft_consensus.cc:2781] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:33:39.643061 13749 raft_consensus.cc:686] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 4 LEADER]: Becoming Leader. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:39.643841 13749 consensus_queue.cc:237] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:39.650530 13313 catalog_manager.cc:5495] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 reported cstate change: term changed from 3 to 4, leader changed from 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131) to 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130). New cstate: current_term: 4 leader_uuid: "9819c221b89b4b3c829980ffc65b3679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:39.940102 13421 raft_consensus.cc:1261] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:39.941650 13758 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:33:39.951246 13577 raft_consensus.cc:1261] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:39.952682 13749 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:39.968880 13577 raft_consensus.cc:1261] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:39.969921 13758 consensus_queue.cc:1035] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:39.978271 13421 raft_consensus.cc:1261] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:39.979439 13749 consensus_queue.cc:1035] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:40.046236 13421 raft_consensus.cc:1261] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:40.047549 13758 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:40.049319 13421 raft_consensus.cc:1261] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:40.050559 13758 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:33:40.058857 13577 raft_consensus.cc:1261] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:40.060088 13749 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:40.070829 13577 raft_consensus.cc:1261] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:40.072027 13758 consensus_queue.cc:1035] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:33:40.115037 13421 raft_consensus.cc:1261] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:40.116223 13749 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:33:40.126981 13577 raft_consensus.cc:1261] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Refusing update from remote peer 9819c221b89b4b3c829980ffc65b3679: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:40.128403 13758 consensus_queue.cc:1035] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:33:42.343191 13282 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260806 20:33:42.344058 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:42.350414 13500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:38946
I20260806 20:33:42.351118 13500 raft_consensus.cc:595] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Received request to transfer leadership to TS 6430856814214be2be901e26e0c9ae73
I20260806 20:33:42.353610 13500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:38946
I20260806 20:33:42.354034 13500 raft_consensus.cc:595] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Received request to transfer leadership to TS 038a7b6ab86f465884b0861c91d19d36
I20260806 20:33:42.356349 13500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:38946
I20260806 20:33:42.356904 13500 raft_consensus.cc:595] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Received request to transfer leadership to TS 6430856814214be2be901e26e0c9ae73
I20260806 20:33:42.358832 13500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:38946
I20260806 20:33:42.359225 13500 raft_consensus.cc:595] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Received request to transfer leadership to TS 6430856814214be2be901e26e0c9ae73
I20260806 20:33:42.361374 13500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8"
dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
mode: GRACEFUL
new_leader_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:38946
I20260806 20:33:42.361827 13500 raft_consensus.cc:595] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Received request to transfer leadership to TS 038a7b6ab86f465884b0861c91d19d36
I20260806 20:33:42.362814 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:33:42.363323 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:42.516140 13758 raft_consensus.cc:979] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:42.517841 13775 raft_consensus.cc:1067] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Signalling peer 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:42.518332 13777 raft_consensus.cc:979] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:42.518913 13776 raft_consensus.cc:1067] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Signalling peer 038a7b6ab86f465884b0861c91d19d36 to start an election
I20260806 20:33:42.519636 13579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:48008
I20260806 20:33:42.520154 13579 raft_consensus.cc:482] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:42.520567 13579 raft_consensus.cc:3037] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:42.520658 13577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8"
dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
from {username='slave'} at 127.0.0.1:48008
I20260806 20:33:42.521168 13577 raft_consensus.cc:482] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:42.521423 13577 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:42.526237 13579 raft_consensus.cc:504] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.526137 13577 raft_consensus.cc:504] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.528381 13577 leader_election.cc:288] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 3 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:42.528517 13579 leader_election.cc:288] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 2 election: Requested vote from peers 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137), 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879)
I20260806 20:33:42.529868 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:42.530349 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:42.530449 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36040d25e4a4103a76ea1099f0c25e8" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6430856814214be2be901e26e0c9ae73"
I20260806 20:33:42.530349 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647a1ce4dc1c4b08b438a2a46d716cb6" candidate_uuid: "038a7b6ab86f465884b0861c91d19d36" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:42.531081 13421 raft_consensus.cc:3037] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:42.531196 13422 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:42.531265 13501 raft_consensus.cc:3032] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:42.531659 13501 raft_consensus.cc:729] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:42.532420 13501 consensus_queue.cc:260] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.533710 13501 raft_consensus.cc:3037] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:42.530719 13500 raft_consensus.cc:3032] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:33:42.534711 13500 raft_consensus.cc:729] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:42.535431 13500 consensus_queue.cc:260] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.536520 13500 raft_consensus.cc:3037] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:42.536713 13422 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 3.
I20260806 20:33:42.538101 13542 leader_election.cc:302] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:42.538268 13421 raft_consensus.cc:2454] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 2.
I20260806 20:33:42.539160 13756 raft_consensus.cc:2781] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:33:42.540345 13501 raft_consensus.cc:2454] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 2.
I20260806 20:33:42.540504 13542 leader_election.cc:302] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:42.543054 13500 raft_consensus.cc:2454] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038a7b6ab86f465884b0861c91d19d36 in term 3.
I20260806 20:33:42.543254 13779 raft_consensus.cc:2781] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:42.543975 13779 raft_consensus.cc:686] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 2 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:42.544625 13756 raft_consensus.cc:686] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Becoming Leader. State: Replica: 038a7b6ab86f465884b0861c91d19d36, State: Running, Role: LEADER
I20260806 20:33:42.545657 13779 consensus_queue.cc:237] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.545670 13756 consensus_queue.cc:237] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.554553 13313 catalog_manager.cc:5495] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 2 to 3, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 3 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:42.570215 13313 catalog_manager.cc:5495] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 reported cstate change: term changed from 1 to 2, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131). New cstate: current_term: 2 leader_uuid: "038a7b6ab86f465884b0861c91d19d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:42.574199 13776 raft_consensus.cc:979] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:42.574782 13775 raft_consensus.cc:1067] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Signalling peer 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:42.576417 13422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:40526
I20260806 20:33:42.577064 13422 raft_consensus.cc:482] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:42.577425 13422 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:42.582170 13422 raft_consensus.cc:504] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.583818 13422 leader_election.cc:288] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 4 election: Requested vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:42.584898 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:42.585217 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25ceb6a22484feb825ea23218a15ac5" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:42.585383 13500 raft_consensus.cc:3032] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:33:42.585671 13500 raft_consensus.cc:729] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:42.585723 13577 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:42.586334 13500 consensus_queue.cc:260] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.587220 13500 raft_consensus.cc:3037] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:42.589999 13577 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 4.
I20260806 20:33:42.591239 13383 leader_election.cc:302] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:42.591493 13500 raft_consensus.cc:2454] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 4.
I20260806 20:33:42.593098 13787 raft_consensus.cc:2781] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:33:42.594182 13787 raft_consensus.cc:686] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 4 LEADER]: Becoming Leader. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:42.595022 13787 consensus_queue.cc:237] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.597380 13776 raft_consensus.cc:979] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:42.597841 13775 raft_consensus.cc:1067] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Signalling peer 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:42.599632 13422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:40526
I20260806 20:33:42.600230 13422 raft_consensus.cc:482] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:42.600641 13422 raft_consensus.cc:3037] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:42.602488 13313 catalog_manager.cc:5495] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 reported cstate change: term changed from 3 to 4, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 6430856814214be2be901e26e0c9ae73 (127.12.248.129). New cstate: current_term: 4 leader_uuid: "6430856814214be2be901e26e0c9ae73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:42.605743 13422 raft_consensus.cc:504] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.608338 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:42.608639 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cfa0f2971548ca8daf530a5b7d6e93" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:42.608932 13500 raft_consensus.cc:3032] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:42.609169 13577 raft_consensus.cc:3037] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:42.609265 13500 raft_consensus.cc:729] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:42.610085 13500 consensus_queue.cc:260] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.611594 13500 raft_consensus.cc:3037] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:42.615649 13577 raft_consensus.cc:2454] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:42.607575 13422 leader_election.cc:288] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Requested vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:42.616637 13383 leader_election.cc:302] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:42.617393 13787 raft_consensus.cc:2781] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:42.617825 13787 raft_consensus.cc:686] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Becoming Leader. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:42.617901 13500 raft_consensus.cc:2454] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 2.
I20260806 20:33:42.618796 13787 consensus_queue.cc:237] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.625028 13313 catalog_manager.cc:5495] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 reported cstate change: term changed from 1 to 2, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 6430856814214be2be901e26e0c9ae73 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "6430856814214be2be901e26e0c9ae73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:42.679841 13776 raft_consensus.cc:979] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: : Instructing follower 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:42.680265 13775 raft_consensus.cc:1067] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Signalling peer 6430856814214be2be901e26e0c9ae73 to start an election
I20260806 20:33:42.681586 13422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0"
dest_uuid: "6430856814214be2be901e26e0c9ae73"
from {username='slave'} at 127.0.0.1:40526
I20260806 20:33:42.681963 13422 raft_consensus.cc:482] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:42.682193 13422 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:42.686195 13422 raft_consensus.cc:504] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.687565 13422 leader_election.cc:288] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 4 election: Requested vote from peers 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130:46879), 038a7b6ab86f465884b0861c91d19d36 (127.12.248.131:37989)
I20260806 20:33:42.688290 13500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9819c221b89b4b3c829980ffc65b3679"
I20260806 20:33:42.688611 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a9c71cc716412683f508e467bb11e0" candidate_uuid: "6430856814214be2be901e26e0c9ae73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "038a7b6ab86f465884b0861c91d19d36"
I20260806 20:33:42.688846 13500 raft_consensus.cc:3032] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:33:42.689131 13577 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:42.689155 13500 raft_consensus.cc:729] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9819c221b89b4b3c829980ffc65b3679, State: Running, Role: LEADER
I20260806 20:33:42.689924 13500 consensus_queue.cc:260] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.690865 13500 raft_consensus.cc:3037] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:33:42.693344 13577 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 4.
I20260806 20:33:42.694204 13383 leader_election.cc:302] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038a7b6ab86f465884b0861c91d19d36, 6430856814214be2be901e26e0c9ae73; no voters:
I20260806 20:33:42.694870 13787 raft_consensus.cc:2781] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:33:42.695227 13787 raft_consensus.cc:686] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 4 LEADER]: Becoming Leader. State: Replica: 6430856814214be2be901e26e0c9ae73, State: Running, Role: LEADER
I20260806 20:33:42.695261 13500 raft_consensus.cc:2454] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6430856814214be2be901e26e0c9ae73 in term 4.
I20260806 20:33:42.696029 13787 consensus_queue.cc:237] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } }
I20260806 20:33:42.703917 13313 catalog_manager.cc:5495] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 reported cstate change: term changed from 3 to 4, leader changed from 9819c221b89b4b3c829980ffc65b3679 (127.12.248.130) to 6430856814214be2be901e26e0c9ae73 (127.12.248.129). New cstate: current_term: 4 leader_uuid: "6430856814214be2be901e26e0c9ae73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:42.926949 13422 raft_consensus.cc:1261] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:42.928107 13779 consensus_queue.cc:1035] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:42.938289 13500 raft_consensus.cc:1261] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:42.940193 13779 consensus_queue.cc:1035] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:42.983619 13422 raft_consensus.cc:1261] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:42.984884 13779 consensus_queue.cc:1035] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6430856814214be2be901e26e0c9ae73" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:42.993564 13500 raft_consensus.cc:1261] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Refusing update from remote peer 038a7b6ab86f465884b0861c91d19d36: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:42.994773 13756 consensus_queue.cc:1035] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:43.024583 13577 raft_consensus.cc:1261] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:43.025751 13787 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:33:43.034435 13500 raft_consensus.cc:1261] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:43.037519 13801 consensus_queue.cc:1035] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:33:43.079838 13577 raft_consensus.cc:1261] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:43.080935 13801 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:33:43.089902 13500 raft_consensus.cc:1261] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:33:43.091080 13788 consensus_queue.cc:1035] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:33:43.103705 13577 raft_consensus.cc:1261] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:43.104935 13788 consensus_queue.cc:1035] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038a7b6ab86f465884b0861c91d19d36" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:43.114140 13500 raft_consensus.cc:1261] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Refusing update from remote peer 6430856814214be2be901e26e0c9ae73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:43.117884 13788 consensus_queue.cc:1035] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9819c221b89b4b3c829980ffc65b3679" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:43.364166 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:43.369081 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:43.369537 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:44.370390 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:44.375205 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:44.375619 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:45.376355 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:45.381381 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:45.381940 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:46.382794 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:46.387755 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:46.388197 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:47.389037 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:47.394101 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:47.394519 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:48.395345 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:48.400032 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:48.400476 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:49.401350 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:49.407217 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:49.407644 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:50.408515 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:50.413367 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:50.413806 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:51.414680 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:51.419380 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:51.419822 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:52.448525 13282 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260806 20:33:52.449709 13282 tablet_server.cc:178] TabletServer@127.12.248.129:0 shutting down...
I20260806 20:33:52.480907 13282 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:33:52.481768 13282 tablet_replica.cc:330] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.482612 13282 raft_consensus.cc:2229] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.483618 13282 raft_consensus.cc:2258] T 04a9c71cc716412683f508e467bb11e0 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.485985 13282 tablet_replica.cc:330] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.486521 13282 raft_consensus.cc:2229] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.487354 13282 raft_consensus.cc:2258] T f25ceb6a22484feb825ea23218a15ac5 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.489023 13282 tablet_replica.cc:330] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.489619 13282 raft_consensus.cc:2229] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.490182 13282 raft_consensus.cc:2258] T dcbd7ac041ae408bbac21f2768bf5279 P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.491864 13282 tablet_replica.cc:330] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.492328 13282 raft_consensus.cc:2229] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.492763 13282 raft_consensus.cc:2258] T 42329950ce5e4e14baf56b8caa9710f1 P 6430856814214be2be901e26e0c9ae73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.494310 13282 tablet_replica.cc:330] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.494776 13282 raft_consensus.cc:2229] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.495177 13282 raft_consensus.cc:2258] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.496763 13282 tablet_replica.cc:330] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.497305 13282 raft_consensus.cc:2229] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.498522 13282 raft_consensus.cc:2258] T e2cfa0f2971548ca8daf530a5b7d6e93 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.500519 13282 tablet_replica.cc:330] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.500994 13282 raft_consensus.cc:2229] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.501461 13282 raft_consensus.cc:2258] T e0a9ed1966eb4cc6968fb4e1b419e47a P 6430856814214be2be901e26e0c9ae73 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.503129 13282 tablet_replica.cc:330] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.503558 13282 raft_consensus.cc:2229] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.503962 13282 raft_consensus.cc:2258] T f36040d25e4a4103a76ea1099f0c25e8 P 6430856814214be2be901e26e0c9ae73 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.505555 13282 tablet_replica.cc:330] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73: stopping tablet replica
I20260806 20:33:52.506023 13282 raft_consensus.cc:2229] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.506485 13282 raft_consensus.cc:2258] T c6eae037fbe24283b5b6bfe3a6c9c699 P 6430856814214be2be901e26e0c9ae73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.617393 13282 tablet_server.cc:194] TabletServer@127.12.248.129:0 shutdown complete.
W20260806 20:33:52.620076 13464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.129:45137: connect: Connection refused (error 111)
W20260806 20:33:52.629599 13542 consensus_peers.cc:487] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 -> Peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Couldn't send request to peer 6430856814214be2be901e26e0c9ae73. Status: Network error: Client connection negotiation failed: client connection to 127.12.248.129:45137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260806 20:33:52.629700 13464 consensus_peers.cc:487] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 -> Peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Couldn't send request to peer 6430856814214be2be901e26e0c9ae73. Status: Network error: Client connection negotiation failed: client connection to 127.12.248.129:45137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260806 20:33:52.630303 13464 consensus_peers.cc:487] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 -> Peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Couldn't send request to peer 6430856814214be2be901e26e0c9ae73. Status: Network error: Client connection negotiation failed: client connection to 127.12.248.129:45137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:33:52.643182 13282 tablet_server.cc:178] TabletServer@127.12.248.130:0 shutting down...
I20260806 20:33:52.671824 13282 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:33:52.672425 13282 tablet_replica.cc:330] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.673130 13282 raft_consensus.cc:2229] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.673663 13282 raft_consensus.cc:2258] T 04a9c71cc716412683f508e467bb11e0 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.675602 13282 tablet_replica.cc:330] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.676072 13282 raft_consensus.cc:2229] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.676589 13282 raft_consensus.cc:2258] T f25ceb6a22484feb825ea23218a15ac5 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.678108 13282 tablet_replica.cc:330] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.678586 13282 raft_consensus.cc:2229] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.679003 13282 raft_consensus.cc:2258] T f36040d25e4a4103a76ea1099f0c25e8 P 9819c221b89b4b3c829980ffc65b3679 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.680663 13282 tablet_replica.cc:330] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.681145 13282 raft_consensus.cc:2229] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.681931 13282 raft_consensus.cc:2258] T c6eae037fbe24283b5b6bfe3a6c9c699 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.683490 13282 tablet_replica.cc:330] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.683941 13282 raft_consensus.cc:2229] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.684324 13282 raft_consensus.cc:2258] T 42329950ce5e4e14baf56b8caa9710f1 P 9819c221b89b4b3c829980ffc65b3679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.685818 13282 tablet_replica.cc:330] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.686340 13282 raft_consensus.cc:2229] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.687048 13282 raft_consensus.cc:2258] T dcbd7ac041ae408bbac21f2768bf5279 P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.688437 13282 tablet_replica.cc:330] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.688941 13282 raft_consensus.cc:2229] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.689648 13282 raft_consensus.cc:2258] T e0a9ed1966eb4cc6968fb4e1b419e47a P 9819c221b89b4b3c829980ffc65b3679 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.691325 13282 tablet_replica.cc:330] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.691725 13282 raft_consensus.cc:2229] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.692075 13282 raft_consensus.cc:2258] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.693495 13282 tablet_replica.cc:330] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679: stopping tablet replica
I20260806 20:33:52.693887 13282 raft_consensus.cc:2229] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.694232 13282 raft_consensus.cc:2258] T e2cfa0f2971548ca8daf530a5b7d6e93 P 9819c221b89b4b3c829980ffc65b3679 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.720971 13282 tablet_server.cc:194] TabletServer@127.12.248.130:0 shutdown complete.
I20260806 20:33:52.739318 13282 tablet_server.cc:178] TabletServer@127.12.248.131:0 shutting down...
W20260806 20:33:52.745342 13542 consensus_peers.cc:487] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 -> Peer 6430856814214be2be901e26e0c9ae73 (127.12.248.129:45137): Couldn't send request to peer 6430856814214be2be901e26e0c9ae73. Status: Network error: Client connection negotiation failed: client connection to 127.12.248.129:45137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:33:52.767545 13282 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:33:52.768338 13282 tablet_replica.cc:330] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.769030 13282 raft_consensus.cc:2229] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.769517 13282 raft_consensus.cc:2258] T c6eae037fbe24283b5b6bfe3a6c9c699 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.771291 13282 tablet_replica.cc:330] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.771781 13282 raft_consensus.cc:2229] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.772635 13282 raft_consensus.cc:2258] T f36040d25e4a4103a76ea1099f0c25e8 P 038a7b6ab86f465884b0861c91d19d36 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.774155 13282 tablet_replica.cc:330] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.774780 13282 raft_consensus.cc:2229] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.775260 13282 raft_consensus.cc:2258] T f25ceb6a22484feb825ea23218a15ac5 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.776894 13282 tablet_replica.cc:330] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.777365 13282 raft_consensus.cc:2229] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.777768 13282 raft_consensus.cc:2258] T e2cfa0f2971548ca8daf530a5b7d6e93 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.779249 13282 tablet_replica.cc:330] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.779716 13282 raft_consensus.cc:2229] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.780515 13282 raft_consensus.cc:2258] T 647a1ce4dc1c4b08b438a2a46d716cb6 P 038a7b6ab86f465884b0861c91d19d36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.781975 13282 tablet_replica.cc:330] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.782429 13282 raft_consensus.cc:2229] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.782845 13282 raft_consensus.cc:2258] T dcbd7ac041ae408bbac21f2768bf5279 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.784245 13282 tablet_replica.cc:330] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.784737 13282 raft_consensus.cc:2229] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:33:52.785514 13282 raft_consensus.cc:2258] T 42329950ce5e4e14baf56b8caa9710f1 P 038a7b6ab86f465884b0861c91d19d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.787070 13282 tablet_replica.cc:330] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.787511 13282 raft_consensus.cc:2229] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.787911 13282 raft_consensus.cc:2258] T e0a9ed1966eb4cc6968fb4e1b419e47a P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.789477 13282 tablet_replica.cc:330] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36: stopping tablet replica
I20260806 20:33:52.789875 13282 raft_consensus.cc:2229] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:33:52.790314 13282 raft_consensus.cc:2258] T 04a9c71cc716412683f508e467bb11e0 P 038a7b6ab86f465884b0861c91d19d36 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:52.818454 13282 tablet_server.cc:194] TabletServer@127.12.248.131:0 shutdown complete.
I20260806 20:33:52.835474 13282 master.cc:537] Master@127.12.248.190:33869 shutting down...
I20260806 20:33:53.129051 13282 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:33:53.129741 13282 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:33:53.130108 13282 tablet_replica.cc:330] T 00000000000000000000000000000000 P aad7d81155684a0c9b005ce25ca89e20: stopping tablet replica
I20260806 20:33:53.147812 13282 master.cc:552] Master@127.12.248.190:33869 shutdown complete.
038a7b6ab86f465884b0861c91d19d36 1
6430856814214be2be901e26e0c9ae73 4
9819c221b89b4b3c829980ffc65b3679 4
038a7b6ab86f465884b0861c91d19d36 3
6430856814214be2be901e26e0c9ae73 3
9819c221b89b4b3c829980ffc65b3679 3
038a7b6ab86f465884b0861c91d19d36 1
6430856814214be2be901e26e0c9ae73 0
9819c221b89b4b3c829980ffc65b3679 8
038a7b6ab86f465884b0861c91d19d36 3
6430856814214be2be901e26e0c9ae73 3
9819c221b89b4b3c829980ffc65b3679 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31190 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260806 20:33:53.189316 13282 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.248.190:36203
I20260806 20:33:53.190454 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:53.195873 13820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:53.196579 13821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:53.198289 13823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:53.200857 13282 server_base.cc:1034] running on GCE node
I20260806 20:33:53.201944 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:33:53.202162 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:53.202317 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048433202303 us; error 0 us; skew 500 ppm
I20260806 20:33:53.202833 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:53.211287 13282 webserver.cc:460] Webserver started at http://127.12.248.190:46845/ using document root <none> and password file <none>
I20260806 20:33:53.211809 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:53.212015 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:53.212328 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:53.213603 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/master-0-root/instance:
uuid: "78141565cd3f4653a9e0c21f5d20d7fd"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.218334 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:33:53.222592 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.223307 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 20:33:53.223587 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/master-0-root
uuid: "78141565cd3f4653a9e0c21f5d20d7fd"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.223865 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:53.246897 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:53.248188 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:53.287106 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.190:36203
I20260806 20:33:53.287222 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.190:36203 every 8 connection(s)
I20260806 20:33:53.291023 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.300196 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd: Bootstrap starting.
I20260806 20:33:53.304634 13881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:53.308959 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd: No bootstrap required, opened a new log
I20260806 20:33:53.310976 13881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" member_type: VOTER }
I20260806 20:33:53.311447 13881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:53.311620 13881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78141565cd3f4653a9e0c21f5d20d7fd, State: Initialized, Role: FOLLOWER
I20260806 20:33:53.312085 13881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" member_type: VOTER }
I20260806 20:33:53.312525 13881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:33:53.312711 13881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:33:53.312924 13881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:53.316766 13881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" member_type: VOTER }
I20260806 20:33:53.317240 13881 leader_election.cc:302] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78141565cd3f4653a9e0c21f5d20d7fd; no voters:
I20260806 20:33:53.318409 13881 leader_election.cc:288] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:33:53.318773 13884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:53.320181 13884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 1 LEADER]: Becoming Leader. State: Replica: 78141565cd3f4653a9e0c21f5d20d7fd, State: Running, Role: LEADER
I20260806 20:33:53.320909 13884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" member_type: VOTER }
I20260806 20:33:53.321592 13881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:33:53.325752 13886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78141565cd3f4653a9e0c21f5d20d7fd. Latest consensus state: current_term: 1 leader_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" member_type: VOTER } }
I20260806 20:33:53.325943 13885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78141565cd3f4653a9e0c21f5d20d7fd" member_type: VOTER } }
I20260806 20:33:53.326455 13886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [sys.catalog]: This master's current role is: LEADER
I20260806 20:33:53.326738 13885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [sys.catalog]: This master's current role is: LEADER
I20260806 20:33:53.333562 13891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:33:53.338979 13891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:33:53.340116 13282 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 20:33:53.347473 13891 catalog_manager.cc:1347] Generated new cluster ID: 5591d197657140f5a10b75a1b69899d3
I20260806 20:33:53.347747 13891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:33:53.366669 13891 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:33:53.368307 13891 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:33:53.382790 13891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd: Generated new TSK 0
I20260806 20:33:53.383463 13891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:33:53.406783 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:53.413095 13902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:53.414168 13903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:53.416141 13905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:53.416826 13282 server_base.cc:1034] running on GCE node
I20260806 20:33:53.417757 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:33:53.417930 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:53.418131 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048433418099 us; error 0 us; skew 500 ppm
I20260806 20:33:53.418661 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:53.421087 13282 webserver.cc:460] Webserver started at http://127.12.248.129:39721/ using document root <none> and password file <none>
I20260806 20:33:53.421587 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:53.421782 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:53.422051 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:53.423280 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-0-root/instance:
uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.427982 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260806 20:33:53.431116 13911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.431859 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 20:33:53.432111 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-0-root
uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.432507 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:53.460489 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:53.461699 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:53.463088 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:33:53.465322 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:33:53.465492 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.465790 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:33:53.465958 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.506719 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.129:45891
I20260806 20:33:53.506830 13977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.129:45891 every 8 connection(s)
I20260806 20:33:53.511657 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:53.518694 13982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:53.518963 13983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:53.522876 13985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:53.523788 13282 server_base.cc:1034] running on GCE node
I20260806 20:33:53.525693 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:33:53.525938 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:53.526155 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048433526138 us; error 0 us; skew 500 ppm
I20260806 20:33:53.526513 13978 heartbeater.cc:347] Connected to a master server at 127.12.248.190:36203
I20260806 20:33:53.526760 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:53.526886 13978 heartbeater.cc:464] Registering TS with master...
I20260806 20:33:53.527774 13978 heartbeater.cc:511] Master 127.12.248.190:36203 requested a full tablet report, sending...
I20260806 20:33:53.529757 13846 ts_manager.cc:194] Registered new tserver with Master: 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891)
I20260806 20:33:53.529914 13282 webserver.cc:460] Webserver started at http://127.12.248.130:35991/ using document root <none> and password file <none>
I20260806 20:33:53.530495 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:53.530697 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:53.530982 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:53.531921 13846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45472
I20260806 20:33:53.532320 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-1-root/instance:
uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.537228 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 20:33:53.540428 13990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.541355 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 20:33:53.541652 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-1-root
uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.541944 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:53.560190 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:53.561483 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:53.562891 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:33:53.565119 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:33:53.565338 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.565579 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:33:53.565744 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.605911 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.130:38575
I20260806 20:33:53.606070 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.130:38575 every 8 connection(s)
I20260806 20:33:53.610502 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:33:53.617677 14057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:33:53.620286 14058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:53.623039 13282 server_base.cc:1034] running on GCE node
W20260806 20:33:53.624408 14060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:33:53.625393 14054 heartbeater.cc:347] Connected to a master server at 127.12.248.190:36203
I20260806 20:33:53.625536 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260806 20:33:53.625751 14054 heartbeater.cc:464] Registering TS with master...
W20260806 20:33:53.625845 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:33:53.626145 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048433626132 us; error 0 us; skew 500 ppm
I20260806 20:33:53.626588 14054 heartbeater.cc:511] Master 127.12.248.190:36203 requested a full tablet report, sending...
I20260806 20:33:53.626701 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:33:53.628744 13846 ts_manager.cc:194] Registered new tserver with Master: b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:53.629925 13282 webserver.cc:460] Webserver started at http://127.12.248.131:33415/ using document root <none> and password file <none>
I20260806 20:33:53.630344 13846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45484
I20260806 20:33:53.630436 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:33:53.630671 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:33:53.630936 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:33:53.632288 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-2-root/instance:
uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.636994 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:33:53.640072 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.640856 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 20:33:53.641106 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-2-root
uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
format_stamp: "Formatted at 2026-08-06 20:33:53 on dist-test-slave-10pc"
I20260806 20:33:53.641357 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786048401793185-13282-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:33:53.653813 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:33:53.654933 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:33:53.656237 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:33:53.658419 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:33:53.658617 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.658861 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:33:53.659009 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:33:53.697481 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.131:42121
I20260806 20:33:53.697590 14130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.131:42121 every 8 connection(s)
I20260806 20:33:53.712217 14131 heartbeater.cc:347] Connected to a master server at 127.12.248.190:36203
I20260806 20:33:53.712633 14131 heartbeater.cc:464] Registering TS with master...
I20260806 20:33:53.713320 14131 heartbeater.cc:511] Master 127.12.248.190:36203 requested a full tablet report, sending...
I20260806 20:33:53.715065 13846 ts_manager.cc:194] Registered new tserver with Master: a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:53.715232 13282 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014632538s
I20260806 20:33:53.716948 13846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45496
I20260806 20:33:53.717687 13282 test_util.cc:268] Using random seed: -831422048
I20260806 20:33:53.742336 13846 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45506:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:33:53.745682 13846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:33:53.814904 14093 tablet_service.cc:1467] Processing CreateTablet for tablet 938f51cb2ffe41f7a9cfe6f99c5f0634 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 20:33:53.816108 14094 tablet_service.cc:1467] Processing CreateTablet for tablet 2e70b403d6754349ab3616a0e32cc2a2 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 20:33:53.817665 14092 tablet_service.cc:1467] Processing CreateTablet for tablet fdc43286cc67445f9d0462d09d2f1c3f (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 20:33:53.818451 14095 tablet_service.cc:1467] Processing CreateTablet for tablet 9221c76981a948cb9aa3ccffbb6843a1 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 20:33:53.814911 14096 tablet_service.cc:1467] Processing CreateTablet for tablet 45e62f8f05af4555ae03bff83152c07d (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 20:33:53.820122 14091 tablet_service.cc:1467] Processing CreateTablet for tablet 703d6b0c741743debb9005ad6b0606aa (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 20:33:53.821600 14093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938f51cb2ffe41f7a9cfe6f99c5f0634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.823137 14090 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d04a9ade94dc8a276327180865e29 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 20:33:53.825836 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d04a9ade94dc8a276327180865e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.826884 14089 tablet_service.cc:1467] Processing CreateTablet for tablet 4283315fbead43cbad6478f9d5233a97 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 20:33:53.828013 14092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc43286cc67445f9d0462d09d2f1c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.828645 14094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e70b403d6754349ab3616a0e32cc2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.829775 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9221c76981a948cb9aa3ccffbb6843a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.831842 14088 tablet_service.cc:1467] Processing CreateTablet for tablet 09da2a02cc3e4ae092b4757a1ebd5b53 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 20:33:53.833292 14088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09da2a02cc3e4ae092b4757a1ebd5b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.832968 14019 tablet_service.cc:1467] Processing CreateTablet for tablet 45e62f8f05af4555ae03bff83152c07d (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 20:33:53.834244 14019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e62f8f05af4555ae03bff83152c07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.835430 14087 tablet_service.cc:1467] Processing CreateTablet for tablet be40a4b3cc7744c99b0a1ecd3f2d60ff (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 20:33:53.836675 14087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40a4b3cc7744c99b0a1ecd3f2d60ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.837302 14018 tablet_service.cc:1467] Processing CreateTablet for tablet 9221c76981a948cb9aa3ccffbb6843a1 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 20:33:53.837934 14096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e62f8f05af4555ae03bff83152c07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.838582 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9221c76981a948cb9aa3ccffbb6843a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.839459 14089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4283315fbead43cbad6478f9d5233a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.841054 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703d6b0c741743debb9005ad6b0606aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.842407 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 2e70b403d6754349ab3616a0e32cc2a2 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 20:33:53.843369 14012 tablet_service.cc:1467] Processing CreateTablet for tablet 4283315fbead43cbad6478f9d5233a97 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 20:33:53.844543 14010 tablet_service.cc:1467] Processing CreateTablet for tablet be40a4b3cc7744c99b0a1ecd3f2d60ff (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 20:33:53.844762 14016 tablet_service.cc:1467] Processing CreateTablet for tablet 938f51cb2ffe41f7a9cfe6f99c5f0634 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 20:33:53.846827 14146 tablet_bootstrap.cc:492] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:53.847023 14011 tablet_service.cc:1467] Processing CreateTablet for tablet 09da2a02cc3e4ae092b4757a1ebd5b53 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 20:33:53.847285 14015 tablet_service.cc:1467] Processing CreateTablet for tablet fdc43286cc67445f9d0462d09d2f1c3f (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 20:33:53.849207 14013 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d04a9ade94dc8a276327180865e29 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 20:33:53.840716 14014 tablet_service.cc:1467] Processing CreateTablet for tablet 703d6b0c741743debb9005ad6b0606aa (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 20:33:53.857578 14146 tablet_bootstrap.cc:654] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:53.869164 14146 tablet_bootstrap.cc:492] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:53.869637 14146 ts_tablet_manager.cc:1397] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.005s
I20260806 20:33:53.870496 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e70b403d6754349ab3616a0e32cc2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.871981 14012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4283315fbead43cbad6478f9d5233a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.872311 14146 raft_consensus.cc:348] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:53.873072 14146 raft_consensus.cc:374] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:53.873420 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40a4b3cc7744c99b0a1ecd3f2d60ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.873370 14146 raft_consensus.cc:729] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:53.874146 14146 consensus_queue.cc:260] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:53.874836 14013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d04a9ade94dc8a276327180865e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.905124 14015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc43286cc67445f9d0462d09d2f1c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.921538 13940 tablet_service.cc:1467] Processing CreateTablet for tablet 9221c76981a948cb9aa3ccffbb6843a1 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 20:33:53.922870 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9221c76981a948cb9aa3ccffbb6843a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.925235 13933 tablet_service.cc:1467] Processing CreateTablet for tablet 09da2a02cc3e4ae092b4757a1ebd5b53 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 20:33:53.925697 13939 tablet_service.cc:1467] Processing CreateTablet for tablet 2e70b403d6754349ab3616a0e32cc2a2 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 20:33:53.926553 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09da2a02cc3e4ae092b4757a1ebd5b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.927950 13938 tablet_service.cc:1467] Processing CreateTablet for tablet 938f51cb2ffe41f7a9cfe6f99c5f0634 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 20:33:53.929258 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938f51cb2ffe41f7a9cfe6f99c5f0634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.934207 13937 tablet_service.cc:1467] Processing CreateTablet for tablet fdc43286cc67445f9d0462d09d2f1c3f (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 20:33:53.936486 13936 tablet_service.cc:1467] Processing CreateTablet for tablet 703d6b0c741743debb9005ad6b0606aa (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 20:33:53.938671 13935 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d04a9ade94dc8a276327180865e29 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 20:33:53.919973 13941 tablet_service.cc:1467] Processing CreateTablet for tablet 45e62f8f05af4555ae03bff83152c07d (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 20:33:53.941099 14014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703d6b0c741743debb9005ad6b0606aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.941576 13934 tablet_service.cc:1467] Processing CreateTablet for tablet 4283315fbead43cbad6478f9d5233a97 (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 20:33:53.942369 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e70b403d6754349ab3616a0e32cc2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.956037 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703d6b0c741743debb9005ad6b0606aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.956693 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4283315fbead43cbad6478f9d5233a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.960063 13941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e62f8f05af4555ae03bff83152c07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.964745 14011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09da2a02cc3e4ae092b4757a1ebd5b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.966516 13932 tablet_service.cc:1467] Processing CreateTablet for tablet be40a4b3cc7744c99b0a1ecd3f2d60ff (DEFAULT_TABLE table=test-workload [id=ac41f1453eba4c808024adb0d4402e4e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 20:33:53.972797 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc43286cc67445f9d0462d09d2f1c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.976794 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40a4b3cc7744c99b0a1ecd3f2d60ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.980969 13935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d04a9ade94dc8a276327180865e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:53.994215 14146 ts_tablet_manager.cc:1428] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.124s user 0.022s sys 0.036s
I20260806 20:33:53.999694 14131 heartbeater.cc:503] Master 127.12.248.190:36203 was elected leader, sending a full tablet report...
I20260806 20:33:54.001762 14016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938f51cb2ffe41f7a9cfe6f99c5f0634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:33:54.002714 14149 tablet_bootstrap.cc:492] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.010489 14150 tablet_bootstrap.cc:492] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.017427 14150 tablet_bootstrap.cc:654] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.024955 14146 tablet_bootstrap.cc:492] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.039860 14149 tablet_bootstrap.cc:654] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.044713 14146 tablet_bootstrap.cc:654] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.047281 14150 tablet_bootstrap.cc:492] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.047756 14150 ts_tablet_manager.cc:1397] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.022s
I20260806 20:33:54.050858 14150 raft_consensus.cc:348] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.051882 14150 raft_consensus.cc:374] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.052253 14150 raft_consensus.cc:729] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.053233 14150 consensus_queue.cc:260] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.055768 13978 heartbeater.cc:503] Master 127.12.248.190:36203 was elected leader, sending a full tablet report...
I20260806 20:33:54.056562 14150 ts_tablet_manager.cc:1428] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20260806 20:33:54.057727 14150 tablet_bootstrap.cc:492] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.064587 14149 tablet_bootstrap.cc:492] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.065097 14149 ts_tablet_manager.cc:1397] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.000s
I20260806 20:33:54.065780 14150 tablet_bootstrap.cc:654] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.069634 14149 raft_consensus.cc:348] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.070432 14149 raft_consensus.cc:374] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.070717 14149 raft_consensus.cc:729] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.076877 14149 consensus_queue.cc:260] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.078410 14146 tablet_bootstrap.cc:492] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.078859 14146 ts_tablet_manager.cc:1397] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.070s user 0.019s sys 0.018s
I20260806 20:33:54.081571 14146 raft_consensus.cc:348] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.082119 14150 tablet_bootstrap.cc:492] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.082300 14146 raft_consensus.cc:374] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.082595 14150 ts_tablet_manager.cc:1397] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20260806 20:33:54.082650 14146 raft_consensus.cc:729] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.083369 14054 heartbeater.cc:503] Master 127.12.248.190:36203 was elected leader, sending a full tablet report...
I20260806 20:33:54.083547 14146 consensus_queue.cc:260] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.085507 14146 ts_tablet_manager.cc:1428] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 20:33:54.085382 14150 raft_consensus.cc:348] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.086107 14150 raft_consensus.cc:374] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.086478 14146 tablet_bootstrap.cc:492] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.087872 14149 ts_tablet_manager.cc:1428] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20260806 20:33:54.088853 14149 tablet_bootstrap.cc:492] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.092881 14146 tablet_bootstrap.cc:654] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.094602 14149 tablet_bootstrap.cc:654] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.086412 14150 raft_consensus.cc:729] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.095904 14150 consensus_queue.cc:260] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.098157 14150 ts_tablet_manager.cc:1428] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260806 20:33:54.099094 14150 tablet_bootstrap.cc:492] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.100014 14149 tablet_bootstrap.cc:492] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.101878 14149 ts_tablet_manager.cc:1397] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260806 20:33:54.104983 14150 tablet_bootstrap.cc:654] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.104585 14149 raft_consensus.cc:348] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.105400 14149 raft_consensus.cc:374] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.105703 14149 raft_consensus.cc:729] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.106621 14146 tablet_bootstrap.cc:492] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.106398 14149 consensus_queue.cc:260] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.107076 14146 ts_tablet_manager.cc:1397] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20260806 20:33:54.108604 14149 ts_tablet_manager.cc:1428] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 20:33:54.109455 14149 tablet_bootstrap.cc:492] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.109740 14146 raft_consensus.cc:348] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.110456 14146 raft_consensus.cc:374] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.110756 14146 raft_consensus.cc:729] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.111441 14146 consensus_queue.cc:260] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.113713 14146 ts_tablet_manager.cc:1428] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 20:33:54.114593 14146 tablet_bootstrap.cc:492] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.115669 14149 tablet_bootstrap.cc:654] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.121526 14146 tablet_bootstrap.cc:654] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.123521 14153 raft_consensus.cc:482] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.123917 14153 raft_consensus.cc:504] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.127398 14146 tablet_bootstrap.cc:492] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.127868 14146 ts_tablet_manager.cc:1397] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260806 20:33:54.128458 14149 tablet_bootstrap.cc:492] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.128922 14149 ts_tablet_manager.cc:1397] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.014s
I20260806 20:33:54.129005 14153 leader_election.cc:288] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.130451 14146 raft_consensus.cc:348] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.131191 14146 raft_consensus.cc:374] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.131472 14146 raft_consensus.cc:729] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.131431 14149 raft_consensus.cc:348] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.132212 14149 raft_consensus.cc:374] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.132543 14149 raft_consensus.cc:729] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.132390 14146 consensus_queue.cc:260] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.133340 14149 consensus_queue.cc:260] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.134713 14146 ts_tablet_manager.cc:1428] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260806 20:33:54.135708 14146 tablet_bootstrap.cc:492] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.141777 14146 tablet_bootstrap.cc:654] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.142894 14149 ts_tablet_manager.cc:1428] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.014s user 0.001s sys 0.006s
I20260806 20:33:54.143893 14149 tablet_bootstrap.cc:492] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.151639 14149 tablet_bootstrap.cc:654] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.158597 14150 tablet_bootstrap.cc:492] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.159147 14150 ts_tablet_manager.cc:1397] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.001s
I20260806 20:33:54.161914 14150 raft_consensus.cc:348] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.162675 14150 raft_consensus.cc:374] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.162992 14150 raft_consensus.cc:729] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.163873 14150 consensus_queue.cc:260] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.164858 14149 tablet_bootstrap.cc:492] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.165500 14149 ts_tablet_manager.cc:1397] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.016s
I20260806 20:33:54.166338 14150 ts_tablet_manager.cc:1428] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:33:54.167361 14150 tablet_bootstrap.cc:492] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.168834 14149 raft_consensus.cc:348] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.169811 14149 raft_consensus.cc:374] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.170171 14149 raft_consensus.cc:729] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.171258 14149 consensus_queue.cc:260] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.176201 14150 tablet_bootstrap.cc:654] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.182579 14149 ts_tablet_manager.cc:1428] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20260806 20:33:54.192298 14149 tablet_bootstrap.cc:492] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.192553 14150 tablet_bootstrap.cc:492] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.193166 14150 ts_tablet_manager.cc:1397] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.009s
I20260806 20:33:54.196322 14146 tablet_bootstrap.cc:492] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.195962 14150 raft_consensus.cc:348] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.196841 14146 ts_tablet_manager.cc:1397] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.061s user 0.025s sys 0.035s
I20260806 20:33:54.196990 14150 raft_consensus.cc:374] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.197448 14150 raft_consensus.cc:729] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.198441 14149 tablet_bootstrap.cc:654] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.198359 14150 consensus_queue.cc:260] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.199648 14146 raft_consensus.cc:348] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.200387 14146 raft_consensus.cc:374] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.200700 14146 raft_consensus.cc:729] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.201583 14146 consensus_queue.cc:260] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.204811 14029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.205768 14150 ts_tablet_manager.cc:1428] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
W20260806 20:33:54.206805 13915 leader_election.cc:341] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.207021 14146 ts_tablet_manager.cc:1428] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260806 20:33:54.207876 14150 tablet_bootstrap.cc:492] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.208213 14146 tablet_bootstrap.cc:492] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.216753 14149 tablet_bootstrap.cc:492] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.216831 14146 tablet_bootstrap.cc:654] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.217298 14149 ts_tablet_manager.cc:1397] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.011s
I20260806 20:33:54.220124 14149 raft_consensus.cc:348] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.221102 14149 raft_consensus.cc:374] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.221680 14149 raft_consensus.cc:729] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.222064 14150 tablet_bootstrap.cc:654] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.224509 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.223116 14149 consensus_queue.cc:260] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.225288 14106 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 0.
I20260806 20:33:54.228919 13914 leader_election.cc:302] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters: b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.230098 14153 raft_consensus.cc:2781] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.230449 14153 raft_consensus.cc:482] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.230782 14153 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.230893 14146 tablet_bootstrap.cc:492] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.231400 14146 ts_tablet_manager.cc:1397] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20260806 20:33:54.234392 14146 raft_consensus.cc:348] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.234916 14150 tablet_bootstrap.cc:492] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.235093 14146 raft_consensus.cc:374] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.235394 14150 ts_tablet_manager.cc:1397] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.010s
I20260806 20:33:54.235440 14146 raft_consensus.cc:729] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.236243 14146 consensus_queue.cc:260] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.238095 14150 raft_consensus.cc:348] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.238828 14150 raft_consensus.cc:374] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.238987 14146 ts_tablet_manager.cc:1428] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:33:54.239151 14150 raft_consensus.cc:729] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.239858 14150 consensus_queue.cc:260] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.240746 14146 tablet_bootstrap.cc:492] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.244732 14156 raft_consensus.cc:482] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.245271 14156 raft_consensus.cc:504] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.248111 14146 tablet_bootstrap.cc:654] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.254634 14156 leader_election.cc:288] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121), 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891)
I20260806 20:33:54.257428 14153 raft_consensus.cc:504] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.261518 14029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
W20260806 20:33:54.265363 13915 leader_election.cc:341] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.266674 14149 ts_tablet_manager.cc:1428] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260806 20:33:54.266489 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:54.267241 14106 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.267654 14149 tablet_bootstrap.cc:492] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.270051 14153 leader_election.cc:288] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.271188 14146 tablet_bootstrap.cc:492] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.271660 14146 ts_tablet_manager.cc:1397] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.005s
I20260806 20:33:54.274417 14146 raft_consensus.cc:348] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.278010 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.279531 14105 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 0.
I20260806 20:33:54.280895 13993 leader_election.cc:302] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:54.281733 14156 raft_consensus.cc:2781] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.282035 14156 raft_consensus.cc:482] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.282321 14156 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.283766 14106 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 1.
I20260806 20:33:54.284971 13914 leader_election.cc:302] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters: b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.286353 14153 raft_consensus.cc:2781] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.286767 14153 raft_consensus.cc:686] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:54.287181 14149 tablet_bootstrap.cc:654] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.287770 14153 consensus_queue.cc:237] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.289170 14156 raft_consensus.cc:504] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.292074 14150 ts_tablet_manager.cc:1428] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.056s user 0.020s sys 0.021s
I20260806 20:33:54.292177 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:54.293362 14150 tablet_bootstrap.cc:492] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.293547 14105 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.300155 14150 tablet_bootstrap.cc:654] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.300917 14156 leader_election.cc:288] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 election: Requested vote from peers a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121), 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891)
I20260806 20:33:54.301998 14146 raft_consensus.cc:374] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.302289 14146 raft_consensus.cc:729] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.302925 14146 consensus_queue.cc:260] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.305040 14149 tablet_bootstrap.cc:492] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.305496 14149 ts_tablet_manager.cc:1397] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.001s
I20260806 20:33:54.306066 14105 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 1.
I20260806 20:33:54.307126 13993 leader_election.cc:302] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:54.308195 14149 raft_consensus.cc:348] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.308936 14149 raft_consensus.cc:374] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.309242 14149 raft_consensus.cc:729] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.309903 14156 raft_consensus.cc:2781] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.309985 14149 consensus_queue.cc:260] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.310567 13845 catalog_manager.cc:5495] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:54.314635 14150 tablet_bootstrap.cc:492] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.315099 14150 ts_tablet_manager.cc:1397] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.003s
I20260806 20:33:54.317106 14149 ts_tablet_manager.cc:1428] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260806 20:33:54.317392 14148 raft_consensus.cc:482] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.318035 14149 tablet_bootstrap.cc:492] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.317852 14148 raft_consensus.cc:504] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.318396 14156 raft_consensus.cc:686] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Becoming Leader. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:33:54.319351 14156 consensus_queue.cc:237] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.318004 14146 ts_tablet_manager.cc:1428] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.046s user 0.005s sys 0.001s
I20260806 20:33:54.322356 14146 tablet_bootstrap.cc:492] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
W20260806 20:33:54.334323 13894 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:33:54.324079 14148 leader_election.cc:288] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.342943 14146 tablet_bootstrap.cc:654] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.344079 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:54.343843 14150 raft_consensus.cc:348] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.345227 14150 raft_consensus.cc:374] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.345701 14150 raft_consensus.cc:729] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
W20260806 20:33:54.347223 13992 leader_election.cc:341] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.345650 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:54.347270 14150 consensus_queue.cc:260] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
W20260806 20:33:54.349092 13992 leader_election.cc:341] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.352363 14150 ts_tablet_manager.cc:1428] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.037s user 0.007s sys 0.003s
I20260806 20:33:54.353083 14148 raft_consensus.cc:482] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.353629 14150 tablet_bootstrap.cc:492] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.353607 14148 raft_consensus.cc:504] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.361810 14148 leader_election.cc:288] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.357911 13844 catalog_manager.cc:5495] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130). New cstate: current_term: 1 leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:54.358094 14149 tablet_bootstrap.cc:654] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.370906 14150 tablet_bootstrap.cc:654] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.372288 14146 tablet_bootstrap.cc:492] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.375072 14146 ts_tablet_manager.cc:1397] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.009s
I20260806 20:33:54.376873 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:54.377568 13950 raft_consensus.cc:2454] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 0.
I20260806 20:33:54.378556 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:54.378458 14146 raft_consensus.cc:348] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.379148 14146 raft_consensus.cc:374] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.379400 14146 raft_consensus.cc:729] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.379665 14153 raft_consensus.cc:482] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.380095 14153 raft_consensus.cc:504] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.380113 14146 consensus_queue.cc:260] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.382154 14153 leader_election.cc:288] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.384234 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.384495 14029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.384847 14146 ts_tablet_manager.cc:1428] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 20:33:54.385143 14105 raft_consensus.cc:2454] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 0.
I20260806 20:33:54.385738 14146 tablet_bootstrap.cc:492] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.386735 13914 leader_election.cc:302] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:33:54.387043 14068 leader_election.cc:302] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:33:54.385154 14029 raft_consensus.cc:2454] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 0.
W20260806 20:33:54.387923 14068 leader_election.cc:341] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.388859 14148 raft_consensus.cc:2781] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.389209 14148 raft_consensus.cc:482] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.389521 14148 raft_consensus.cc:3037] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.389761 14153 raft_consensus.cc:2781] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.390209 14153 raft_consensus.cc:482] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.390616 14153 raft_consensus.cc:3037] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.391258 14146 tablet_bootstrap.cc:654] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.398021 14148 raft_consensus.cc:504] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.399573 14153 raft_consensus.cc:504] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.403793 14149 tablet_bootstrap.cc:492] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.416785 14149 ts_tablet_manager.cc:1397] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.004s
I20260806 20:33:54.419587 14149 raft_consensus.cc:348] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.420321 14149 raft_consensus.cc:374] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.420631 14149 raft_consensus.cc:729] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.421259 14149 consensus_queue.cc:260] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.405794 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:54.407354 14148 leader_election.cc:288] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.410374 14029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.411346 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.424402 14028 raft_consensus.cc:2454] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 0.
W20260806 20:33:54.424748 14070 leader_election.cc:341] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.425326 14149 ts_tablet_manager.cc:1428] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260806 20:33:54.425706 14070 leader_election.cc:302] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.426271 14149 tablet_bootstrap.cc:492] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.426518 14148 raft_consensus.cc:2781] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.426875 14148 raft_consensus.cc:482] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.426952 14156 raft_consensus.cc:482] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.427217 14148 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.427441 14156 raft_consensus.cc:504] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.429513 14156 leader_election.cc:288] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.430526 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.411722 14146 tablet_bootstrap.cc:492] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.412254 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
W20260806 20:33:54.431663 13993 leader_election.cc:341] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.413910 14150 tablet_bootstrap.cc:492] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.423436 13951 raft_consensus.cc:3037] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.432005 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:54.432700 14146 ts_tablet_manager.cc:1397] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.010s
W20260806 20:33:54.433427 13992 leader_election.cc:341] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.433882 13992 leader_election.cc:302] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5
W20260806 20:33:54.435308 14070 leader_election.cc:341] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.435261 14146 raft_consensus.cc:348] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.435990 14156 raft_consensus.cc:2726] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:54.436038 14146 raft_consensus.cc:374] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.436776 14146 raft_consensus.cc:729] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.437278 14149 tablet_bootstrap.cc:654] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.437480 14146 consensus_queue.cc:260] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.437867 14148 raft_consensus.cc:504] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.439601 13951 raft_consensus.cc:2454] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 1.
I20260806 20:33:54.441126 14150 ts_tablet_manager.cc:1397] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.088s user 0.019s sys 0.025s
I20260806 20:33:54.441131 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:54.442755 14068 leader_election.cc:302] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters: b4bdb3cf462e488bb99dca8f2ddf0148
W20260806 20:33:54.445248 14068 leader_election.cc:341] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.444913 14150 raft_consensus.cc:348] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.445951 14150 raft_consensus.cc:374] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.446323 14150 raft_consensus.cc:729] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.447500 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:54.448086 14027 raft_consensus.cc:3037] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.447690 14150 consensus_queue.cc:260] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.451185 14153 leader_election.cc:288] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.452401 14148 leader_election.cc:288] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.453049 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:54.453588 14028 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.452942 14148 raft_consensus.cc:2781] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.456344 14148 raft_consensus.cc:686] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:54.457178 14148 consensus_queue.cc:237] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.458705 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:54.459455 14105 raft_consensus.cc:3037] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.471094 14028 raft_consensus.cc:2454] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 1.
I20260806 20:33:54.471975 14148 raft_consensus.cc:482] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.472311 14070 leader_election.cc:302] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.472435 14148 raft_consensus.cc:504] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.472800 14153 raft_consensus.cc:482] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.473451 14153 raft_consensus.cc:504] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.475386 14029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.475931 14153 leader_election.cc:288] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.476013 14029 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 0.
I20260806 20:33:54.478507 13846 catalog_manager.cc:5495] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:54.479929 14070 leader_election.cc:302] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:54.476532 14148 leader_election.cc:288] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.477914 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.480931 14026 raft_consensus.cc:2454] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 0.
I20260806 20:33:54.481664 14027 raft_consensus.cc:2454] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 1.
I20260806 20:33:54.482758 14193 raft_consensus.cc:2781] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.483227 14193 raft_consensus.cc:686] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:54.484040 14193 consensus_queue.cc:237] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.485993 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:54.486840 13950 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 0.
I20260806 20:33:54.494210 13915 leader_election.cc:302] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:54.495007 14146 ts_tablet_manager.cc:1428] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.062s user 0.022s sys 0.038s
I20260806 20:33:54.495594 14153 raft_consensus.cc:2781] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.495918 14146 tablet_bootstrap.cc:492] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5: Bootstrap starting.
I20260806 20:33:54.495977 14153 raft_consensus.cc:482] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.496490 14153 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.501518 14146 tablet_bootstrap.cc:654] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.489255 14148 raft_consensus.cc:2781] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.502736 14148 raft_consensus.cc:482] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.503180 14148 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.503381 14153 raft_consensus.cc:504] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.506616 14153 leader_election.cc:288] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.492388 14105 raft_consensus.cc:2454] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 1.
I20260806 20:33:54.495854 13915 leader_election.cc:302] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:54.494611 14150 ts_tablet_manager.cc:1428] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.053s user 0.009s sys 0.000s
I20260806 20:33:54.511658 14153 raft_consensus.cc:2781] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.513273 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.514626 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:54.514986 14150 tablet_bootstrap.cc:492] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.514182 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:54.517069 14153 raft_consensus.cc:686] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:54.519591 14148 raft_consensus.cc:504] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.519762 14028 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.522533 14195 raft_consensus.cc:482] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.523146 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:54.523701 13950 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.517539 14105 raft_consensus.cc:2454] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 0.
I20260806 20:33:54.528314 14149 tablet_bootstrap.cc:492] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.528820 14149 ts_tablet_manager.cc:1397] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.103s user 0.008s sys 0.016s
I20260806 20:33:54.531473 14149 raft_consensus.cc:348] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.532150 14149 raft_consensus.cc:374] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.533218 14028 raft_consensus.cc:2454] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 1.
I20260806 20:33:54.534418 14148 leader_election.cc:288] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.534762 13915 leader_election.cc:302] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, b4bdb3cf462e488bb99dca8f2ddf0148; no voters: a6a8d885dee24fe6b9541df33ffc43b5
I20260806 20:33:54.537478 14146 tablet_bootstrap.cc:492] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5: No bootstrap required, opened a new log
I20260806 20:33:54.534792 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:54.522964 14195 raft_consensus.cc:504] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.538208 14146 ts_tablet_manager.cc:1397] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.004s
I20260806 20:33:54.520385 14153 consensus_queue.cc:237] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.538627 14028 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.540727 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:54.541522 13951 raft_consensus.cc:2454] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 0.
I20260806 20:33:54.542936 14146 raft_consensus.cc:348] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.543761 14146 raft_consensus.cc:374] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.544049 14146 raft_consensus.cc:729] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.545740 14068 leader_election.cc:302] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:33:54.536144 14150 tablet_bootstrap.cc:654] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.544749 14146 consensus_queue.cc:260] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.532419 14149 raft_consensus.cc:729] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.537742 14171 raft_consensus.cc:2781] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.547349 14148 raft_consensus.cc:2781] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.547794 14148 raft_consensus.cc:482] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.548233 14148 raft_consensus.cc:3037] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.549149 13846 catalog_manager.cc:5495] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:54.550460 14149 consensus_queue.cc:260] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.552646 14195 leader_election.cc:288] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.552984 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
W20260806 20:33:54.554456 14070 leader_election.cc:341] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.560336 14146 ts_tablet_manager.cc:1428] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
I20260806 20:33:54.558720 14148 raft_consensus.cc:504] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.561642 14149 ts_tablet_manager.cc:1428] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20260806 20:33:54.562750 14149 tablet_bootstrap.cc:492] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.565198 14148 leader_election.cc:288] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.565120 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:54.565125 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
W20260806 20:33:54.567341 14070 leader_election.cc:341] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.567982 14148 raft_consensus.cc:482] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.568470 14148 raft_consensus.cc:504] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.565810 13951 raft_consensus.cc:3037] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.564028 13950 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 1.
I20260806 20:33:54.571148 14068 leader_election.cc:302] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:33:54.575498 14149 tablet_bootstrap.cc:654] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.571751 14171 raft_consensus.cc:686] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:54.574123 14193 raft_consensus.cc:2781] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.577672 14193 raft_consensus.cc:686] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:54.577486 14171 consensus_queue.cc:237] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.580668 14156 raft_consensus.cc:482] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.581142 14156 raft_consensus.cc:504] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.585116 14028 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 1.
I20260806 20:33:54.589146 14156 leader_election.cc:288] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.589809 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:54.589769 14193 consensus_queue.cc:237] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.590353 13949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
W20260806 20:33:54.592301 13992 leader_election.cc:341] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:33:54.592381 14068 leader_election.cc:341] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.592981 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.593797 14105 raft_consensus.cc:2454] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 0.
I20260806 20:33:54.595088 13993 leader_election.cc:302] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.596418 14156 raft_consensus.cc:2781] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.596882 14156 raft_consensus.cc:482] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.597282 14156 raft_consensus.cc:3037] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.602309 14150 tablet_bootstrap.cc:492] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.602794 14150 ts_tablet_manager.cc:1397] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.088s user 0.012s sys 0.022s
I20260806 20:33:54.602937 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.603736 14028 raft_consensus.cc:2454] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 0.
I20260806 20:33:54.605109 14070 leader_election.cc:302] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.605507 14150 raft_consensus.cc:348] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.606117 14195 raft_consensus.cc:2781] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:33:54.604998 13846 catalog_manager.cc:5495] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:54.606544 14195 raft_consensus.cc:482] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:33:54.606906 14195 raft_consensus.cc:3037] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.607189 13951 raft_consensus.cc:2454] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 1.
I20260806 20:33:54.606178 14150 raft_consensus.cc:374] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.608253 14068 leader_election.cc:302] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters: b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.608839 14148 leader_election.cc:288] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.612831 14156 raft_consensus.cc:504] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.613317 14195 raft_consensus.cc:504] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.616053 14191 raft_consensus.cc:2781] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.619015 14150 raft_consensus.cc:729] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.619362 14195 leader_election.cc:288] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.619928 14150 consensus_queue.cc:260] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.621382 14149 tablet_bootstrap.cc:492] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.621799 14149 ts_tablet_manager.cc:1397] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.006s
I20260806 20:33:54.622334 14150 ts_tablet_manager.cc:1428] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260806 20:33:54.622114 13949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:54.623396 14150 tablet_bootstrap.cc:492] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
W20260806 20:33:54.623895 13992 leader_election.cc:341] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:33:54.624910 14149 raft_consensus.cc:348] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.625735 14149 raft_consensus.cc:374] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.626514 14149 raft_consensus.cc:729] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.625741 13845 catalog_manager.cc:5495] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:54.627166 14149 consensus_queue.cc:260] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.627188 13844 catalog_manager.cc:5495] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:54.629258 14149 ts_tablet_manager.cc:1428] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:33:54.629948 14191 raft_consensus.cc:686] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:54.630327 14150 tablet_bootstrap.cc:654] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.630329 14149 tablet_bootstrap.cc:492] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148: Bootstrap starting.
I20260806 20:33:54.630802 14191 consensus_queue.cc:237] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.636698 14149 tablet_bootstrap.cc:654] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.647581 14149 tablet_bootstrap.cc:492] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148: No bootstrap required, opened a new log
I20260806 20:33:54.648005 14149 ts_tablet_manager.cc:1397] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260806 20:33:54.650308 14149 raft_consensus.cc:348] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.650959 14149 raft_consensus.cc:374] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.651252 14149 raft_consensus.cc:729] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.651647 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:54.652230 14028 raft_consensus.cc:3037] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.652030 14149 consensus_queue.cc:260] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.653246 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:54.656708 13951 raft_consensus.cc:3037] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.658862 14149 ts_tablet_manager.cc:1428] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20260806 20:33:54.658964 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:54.659713 14105 raft_consensus.cc:3037] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.666427 13845 catalog_manager.cc:5495] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:54.668138 14028 raft_consensus.cc:2454] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 1.
I20260806 20:33:54.668561 14156 leader_election.cc:288] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.669147 14070 leader_election.cc:302] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:54.670114 14191 raft_consensus.cc:2781] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.670792 14028 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.670929 14105 raft_consensus.cc:2454] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 1.
I20260806 20:33:54.670783 14191 raft_consensus.cc:686] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:54.672235 14191 consensus_queue.cc:237] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.672406 13993 leader_election.cc:302] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.673877 14156 raft_consensus.cc:2781] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:33:54.673909 14171 raft_consensus.cc:482] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.678705 14153 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:54.680321 14171 raft_consensus.cc:504] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:54.683353 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.684288 13951 raft_consensus.cc:2454] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 1.
I20260806 20:33:54.684389 14106 raft_consensus.cc:2379] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in current term 1: Already voted for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in this term.
I20260806 20:33:54.686057 14156 raft_consensus.cc:686] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Becoming Leader. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:33:54.687688 14156 consensus_queue.cc:237] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.689649 14171 leader_election.cc:288] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:54.690688 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.691368 14150 tablet_bootstrap.cc:492] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.691836 14150 ts_tablet_manager.cc:1397] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.044s
I20260806 20:33:54.691700 13915 leader_election.cc:302] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0; no voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.693063 14171 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.694800 14150 raft_consensus.cc:348] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.695708 14150 raft_consensus.cc:374] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.696105 14150 raft_consensus.cc:729] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.697201 14150 consensus_queue.cc:260] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.699473 14150 ts_tablet_manager.cc:1428] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:33:54.700348 14150 tablet_bootstrap.cc:492] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0: Bootstrap starting.
I20260806 20:33:54.711452 14150 tablet_bootstrap.cc:654] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0: Neither blocks nor log segments found. Creating new log.
I20260806 20:33:54.711970 13844 catalog_manager.cc:5495] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 1 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:54.713500 14171 raft_consensus.cc:2726] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:54.717676 14153 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:54.721750 13846 catalog_manager.cc:5495] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130). New cstate: current_term: 1 leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:54.722903 14150 tablet_bootstrap.cc:492] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0: No bootstrap required, opened a new log
I20260806 20:33:54.723335 14150 ts_tablet_manager.cc:1397] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20260806 20:33:54.726051 14156 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:54.725857 14150 raft_consensus.cc:348] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.726847 14150 raft_consensus.cc:374] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:33:54.727205 14150 raft_consensus.cc:729] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Initialized, Role: FOLLOWER
I20260806 20:33:54.727957 14150 consensus_queue.cc:260] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.730371 14150 ts_tablet_manager.cc:1428] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:33:54.744656 14156 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:54.802156 14028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:44500
I20260806 20:33:54.802794 14028 raft_consensus.cc:595] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.819036 14105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:59636
I20260806 20:33:54.819531 14105 raft_consensus.cc:595] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.824095 14105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:59636
I20260806 20:33:54.824620 14105 raft_consensus.cc:595] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.828923 14105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:59636
I20260806 20:33:54.829416 14105 raft_consensus.cc:595] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:54.836355 14105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
mode: GRACEFUL
new_leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:59636
I20260806 20:33:54.836941 14105 raft_consensus.cc:595] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Received request to transfer leadership to TS b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.850435 13951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:50796
I20260806 20:33:54.850962 13951 raft_consensus.cc:595] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Received request to transfer leadership to TS b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.855459 14105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
mode: GRACEFUL
new_leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:59636
I20260806 20:33:54.855945 14105 raft_consensus.cc:595] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Received request to transfer leadership to TS b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.860176 13951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:50796
I20260806 20:33:54.860778 13951 raft_consensus.cc:595] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Received request to transfer leadership to TS a6a8d885dee24fe6b9541df33ffc43b5
I20260806 20:33:54.896432 14191 consensus_queue.cc:1035] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:54.902942 14148 raft_consensus.cc:979] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:54.903573 14204 raft_consensus.cc:1067] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:54.905213 13949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50794
I20260806 20:33:54.906657 13949 raft_consensus.cc:482] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:54.907110 13949 raft_consensus.cc:3037] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:54.907769 14028 raft_consensus.cc:3037] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:54.913885 13949 raft_consensus.cc:504] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.915989 14204 consensus_queue.cc:1035] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:54.916545 13949 leader_election.cc:288] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.917716 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:54.918296 14105 raft_consensus.cc:3032] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:54.918617 14105 raft_consensus.cc:729] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:54.919603 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f51cb2ffe41f7a9cfe6f99c5f0634" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:54.919344 14105 consensus_queue.cc:260] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.920645 14105 raft_consensus.cc:3037] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:54.927645 14105 raft_consensus.cc:2454] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:33:54.928593 13914 leader_election.cc:302] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters: b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.929265 14153 raft_consensus.cc:2781] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:54.929744 14153 raft_consensus.cc:686] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:54.930435 14153 consensus_queue.cc:237] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.937361 13844 catalog_manager.cc:5495] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 1 to 2, leader changed from a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:54.975390 14148 consensus_queue.cc:1035] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:54.975504 14204 consensus_queue.cc:1035] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:54.982769 14148 raft_consensus.cc:979] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:54.983254 14193 raft_consensus.cc:1067] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:54.984081 14193 raft_consensus.cc:979] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5: : Instructing follower b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:33:54.984447 14193 raft_consensus.cc:1067] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Signalling peer b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:33:54.986332 14028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:44488
I20260806 20:33:54.987732 14028 raft_consensus.cc:482] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:54.988114 14028 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:54.989516 14153 raft_consensus.cc:482] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.989971 14153 raft_consensus.cc:504] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:54.991914 14153 leader_election.cc:288] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:54.992830 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:54.993738 13949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50794
I20260806 20:33:54.994302 13949 raft_consensus.cc:482] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:54.994681 13949 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:54.995476 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:54.996928 13914 leader_election.cc:302] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0; no voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:54.998289 14217 raft_consensus.cc:482] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:54.998365 14195 consensus_queue.cc:1035] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:54.998816 14217 raft_consensus.cc:504] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.001046 13951 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:55.002341 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" is_pre_election: true
I20260806 20:33:55.001055 14217 leader_election.cc:288] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:55.003158 13950 raft_consensus.cc:2379] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b4bdb3cf462e488bb99dca8f2ddf0148 in current term 1: Already voted for candidate a6a8d885dee24fe6b9541df33ffc43b5 in this term.
I20260806 20:33:55.003067 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:55.001502 13949 raft_consensus.cc:504] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.005172 13993 leader_election.cc:302] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5
I20260806 20:33:55.006067 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:55.006219 13949 leader_election.cc:288] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:55.006650 14026 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.007205 14217 raft_consensus.cc:3037] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:55.007331 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:55.007921 14106 raft_consensus.cc:3032] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:55.008246 14106 raft_consensus.cc:729] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:55.009042 14106 consensus_queue.cc:260] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.010041 14106 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:54.998310 14028 raft_consensus.cc:504] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.013127 13949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:55.016901 14106 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:33:55.018051 13914 leader_election.cc:302] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:33:55.018183 14026 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:33:55.018930 14171 raft_consensus.cc:2781] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:55.019042 14217 raft_consensus.cc:2726] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:55.019435 14171 raft_consensus.cc:686] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:55.019557 14027 raft_consensus.cc:1226] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Rejecting Update request from peer a6a8d885dee24fe6b9541df33ffc43b5 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260806 20:33:55.020375 14171 consensus_queue.cc:237] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.021108 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:55.022186 14028 leader_election.cc:288] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 2 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:55.021736 14105 raft_consensus.cc:3032] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:55.023330 14105 raft_consensus.cc:729] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:55.024289 14153 raft_consensus.cc:2726] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:55.025825 14199 raft_consensus.cc:482] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:33:55.024761 14105 consensus_queue.cc:260] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.026321 14199 raft_consensus.cc:504] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.027639 14104 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:55.032193 14199 leader_election.cc:288] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:55.027976 14105 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:33:55.028362 14191 consensus_queue.cc:1159] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" responder_term: 1 status { last_received { term: 0 index: 0 } last_committed_idx: 0 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:33:55.038455 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" is_pre_election: true
I20260806 20:33:55.041961 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" is_pre_election: true
I20260806 20:33:55.042776 14106 raft_consensus.cc:2379] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in current term 1: Already voted for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in this term.
I20260806 20:33:55.042371 14199 consensus_queue.cc:1035] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:55.044061 13914 leader_election.cc:302] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0; no voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:33:55.045001 14105 raft_consensus.cc:2454] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 2.
I20260806 20:33:55.046026 13993 leader_election.cc:302] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters: 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:33:55.047334 14217 raft_consensus.cc:2781] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:55.047868 14217 raft_consensus.cc:686] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Becoming Leader. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:33:55.047821 14153 consensus_queue.cc:1035] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.046875 14199 raft_consensus.cc:3037] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:33:55.050275 13845 catalog_manager.cc:5495] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 1 to 2, leader changed from a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:55.052022 14148 consensus_queue.cc:1035] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.048821 14217 consensus_queue.cc:237] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.063179 14199 raft_consensus.cc:2726] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:33:55.065162 13844 catalog_manager.cc:5495] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 reported cstate change: term changed from 1 to 2, leader changed from a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131) to b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130). New cstate: current_term: 2 leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:55.072314 14199 raft_consensus.cc:979] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:33:55.072788 14171 consensus_queue.cc:1035] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.073769 14171 raft_consensus.cc:1067] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Signalling peer a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:33:55.079493 14171 consensus_queue.cc:1035] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.087903 14103 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:59628
I20260806 20:33:55.088523 14103 raft_consensus.cc:482] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:55.088876 14103 raft_consensus.cc:3037] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.091557 14148 consensus_queue.cc:1035] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.095141 14103 raft_consensus.cc:504] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.102334 14103 leader_election.cc:288] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 2 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:55.102432 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:55.103058 14026 raft_consensus.cc:3037] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.104061 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40a4b3cc7744c99b0a1ecd3f2d60ff" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:55.104696 13951 raft_consensus.cc:3032] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:55.105014 13951 raft_consensus.cc:729] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:55.105746 13951 consensus_queue.cc:260] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.107111 13951 raft_consensus.cc:3037] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.108907 14171 raft_consensus.cc:979] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:33:55.109017 14193 raft_consensus.cc:979] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5: : Instructing follower b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:33:55.109408 14171 raft_consensus.cc:1067] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Signalling peer b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:33:55.109432 14193 raft_consensus.cc:1067] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Signalling peer b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:33:55.111037 14027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:44472
I20260806 20:33:55.112243 14028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:44488
I20260806 20:33:55.111590 14027 raft_consensus.cc:482] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:55.113457 14028 raft_consensus.cc:482] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:55.113528 14027 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.113963 14028 raft_consensus.cc:3037] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.111944 14026 raft_consensus.cc:2454] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 2.
I20260806 20:33:55.114367 13951 raft_consensus.cc:2454] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 2.
I20260806 20:33:55.116629 14070 leader_election.cc:302] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:55.117447 14193 raft_consensus.cc:2781] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:55.117897 14193 raft_consensus.cc:686] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:55.118621 14193 consensus_queue.cc:237] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.121519 14027 raft_consensus.cc:504] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.124186 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:55.124857 13951 raft_consensus.cc:3032] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:55.125288 13951 raft_consensus.cc:729] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:55.126011 13951 consensus_queue.cc:260] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.126869 13951 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.129540 14028 raft_consensus.cc:504] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.129717 13846 catalog_manager.cc:5495] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 1 to 2, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 2 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:55.132047 13951 raft_consensus.cc:2454] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 2.
I20260806 20:33:55.132268 13949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:55.132936 13949 raft_consensus.cc:3037] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.133364 13992 leader_election.cc:302] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:55.134157 14217 raft_consensus.cc:2781] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:55.134574 14217 raft_consensus.cc:686] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Becoming Leader. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:33:55.135223 14217 consensus_queue.cc:237] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.132422 14027 leader_election.cc:288] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 2 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:55.139824 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:55.140393 14103 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.141001 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09da2a02cc3e4ae092b4757a1ebd5b53" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:55.141578 14106 raft_consensus.cc:3032] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:55.141885 14106 raft_consensus.cc:729] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:55.142614 14106 consensus_queue.cc:260] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.143658 14106 raft_consensus.cc:3037] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.146531 13846 catalog_manager.cc:5495] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 reported cstate change: term changed from 1 to 2, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130). New cstate: current_term: 2 leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:55.146974 14103 raft_consensus.cc:2454] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 2.
I20260806 20:33:55.147581 14028 leader_election.cc:288] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 2 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:55.150041 14106 raft_consensus.cc:2454] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 2.
I20260806 20:33:55.151180 13993 leader_election.cc:302] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:55.151902 14217 raft_consensus.cc:2781] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:55.152956 14217 raft_consensus.cc:686] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Becoming Leader. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:33:55.153426 13949 raft_consensus.cc:2454] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 2.
I20260806 20:33:55.153760 14217 consensus_queue.cc:237] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.155920 14193 consensus_queue.cc:1035] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.161031 13846 catalog_manager.cc:5495] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 reported cstate change: term changed from 1 to 2, leader changed from a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131) to b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130). New cstate: current_term: 2 leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:55.163398 14148 raft_consensus.cc:979] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:55.163841 14204 raft_consensus.cc:1067] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:55.167980 13951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50794
I20260806 20:33:55.171322 13951 raft_consensus.cc:482] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:55.171271 14217 consensus_queue.cc:1035] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:55.171890 13951 raft_consensus.cc:3037] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.171310 14204 consensus_queue.cc:1035] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.177928 13951 raft_consensus.cc:504] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.182394 13951 leader_election.cc:288] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:33:55.183558 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:55.184983 14103 raft_consensus.cc:3032] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:55.185740 14103 raft_consensus.cc:729] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:55.186592 14103 consensus_queue.cc:260] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.187721 14103 raft_consensus.cc:3037] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.184274 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc43286cc67445f9d0462d09d2f1c3f" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:55.188730 14027 raft_consensus.cc:3037] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.189389 14217 consensus_queue.cc:1035] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.198752 14027 raft_consensus.cc:2454] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:33:55.200742 14103 raft_consensus.cc:2454] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:33:55.202073 13914 leader_election.cc:302] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:33:55.203245 14153 raft_consensus.cc:2781] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:55.203744 14153 raft_consensus.cc:686] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:55.204638 14153 consensus_queue.cc:237] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:55.216842 13845 catalog_manager.cc:5495] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 1 to 2, leader changed from a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:55.219939 14156 raft_consensus.cc:979] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:55.220427 14217 raft_consensus.cc:1067] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:33:55.221874 13951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50792
I20260806 20:33:55.222360 13951 raft_consensus.cc:482] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:55.222711 13951 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.229144 13951 raft_consensus.cc:504] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:55.230767 13951 leader_election.cc:288] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Requested vote from peers a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:55.231576 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:33:55.231854 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:55.232219 14106 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.232422 14027 raft_consensus.cc:3032] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:55.232797 14027 raft_consensus.cc:729] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:33:55.233544 14027 consensus_queue.cc:260] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:55.234534 14027 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.237385 14106 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:33:55.238449 13914 leader_election.cc:302] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:33:55.239351 14153 raft_consensus.cc:2781] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:55.239792 14153 raft_consensus.cc:686] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:55.239796 14027 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:33:55.240732 14153 consensus_queue.cc:237] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:55.247273 13845 catalog_manager.cc:5495] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 1 to 2, leader changed from b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:55.317116 14106 raft_consensus.cc:1261] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.318392 14153 consensus_queue.cc:1035] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.326274 14027 raft_consensus.cc:3037] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.333060 14027 raft_consensus.cc:1261] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.334342 14225 consensus_queue.cc:1035] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.536864 13951 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:55.541725 13951 raft_consensus.cc:1261] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.542987 14217 consensus_queue.cc:1035] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:33:55.548663 13949 raft_consensus.cc:1261] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.550028 14179 consensus_queue.cc:1035] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.552696 14106 raft_consensus.cc:1261] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.554051 14217 consensus_queue.cc:1035] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.571161 14106 raft_consensus.cc:1261] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.574265 14237 consensus_queue.cc:1035] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:33:55.579936 14027 raft_consensus.cc:1261] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.583232 13949 raft_consensus.cc:1261] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.584643 14225 consensus_queue.cc:1035] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:33:55.587074 14217 consensus_queue.cc:1035] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:33:55.592021 14106 raft_consensus.cc:1261] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.594213 14225 consensus_queue.cc:1035] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:33:55.605190 14106 raft_consensus.cc:1261] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.606971 14237 consensus_queue.cc:1035] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.642836 14103 raft_consensus.cc:1261] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.644141 14199 consensus_queue.cc:1035] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.648851 14027 raft_consensus.cc:1261] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.650354 14153 consensus_queue.cc:1035] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.668943 14027 raft_consensus.cc:1261] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.670228 14191 consensus_queue.cc:1035] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.678472 13949 raft_consensus.cc:1261] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.680006 14204 consensus_queue.cc:1035] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.744374 14103 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.745586 14153 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:55.752496 14027 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:55.754169 14171 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:57.888406 13282 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260806 20:33:57.889060 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:57.905159 13949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:46726
I20260806 20:33:57.905686 13949 raft_consensus.cc:595] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Received request to transfer leadership to TS a6a8d885dee24fe6b9541df33ffc43b5
I20260806 20:33:57.908001 13949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:46726
I20260806 20:33:57.908530 13949 raft_consensus.cc:595] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Received request to transfer leadership to TS a6a8d885dee24fe6b9541df33ffc43b5
I20260806 20:33:57.909572 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 20:33:57.909998 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:33:58.064992 14171 raft_consensus.cc:979] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:33:58.065428 14244 raft_consensus.cc:1067] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Signalling peer a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:33:58.066771 14103 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:59628
I20260806 20:33:58.067210 14103 raft_consensus.cc:482] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:58.067448 14103 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:58.073313 14103 raft_consensus.cc:504] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:58.074700 14103 leader_election.cc:288] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 3 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891)
I20260806 20:33:58.075405 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:58.075740 13949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:58.076011 14027 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:58.076301 13949 raft_consensus.cc:3032] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:33:58.076640 13949 raft_consensus.cc:729] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:58.077396 13949 consensus_queue.cc:260] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:58.078626 13949 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:33:58.082751 14027 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 3.
I20260806 20:33:58.083959 14070 leader_election.cc:302] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:58.084734 14239 raft_consensus.cc:2781] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:33:58.085073 13949 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 3.
I20260806 20:33:58.085825 14239 raft_consensus.cc:686] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:58.086722 14239 consensus_queue.cc:237] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:33:58.092962 13846 catalog_manager.cc:5495] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 2 to 3, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 3 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } }
I20260806 20:33:58.216845 14171 raft_consensus.cc:979] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:33:58.217304 14244 raft_consensus.cc:1067] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Signalling peer a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:33:58.218586 14106 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:59628
I20260806 20:33:58.219211 14106 raft_consensus.cc:482] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:33:58.219592 14106 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:58.224655 14106 raft_consensus.cc:504] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:58.226336 14106 leader_election.cc:288] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 2 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:33:58.227319 13949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:33:58.227353 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:33:58.227933 13949 raft_consensus.cc:3032] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:33:58.228012 14027 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:58.228207 13949 raft_consensus.cc:729] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:33:58.228902 13949 consensus_queue.cc:260] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:58.229779 13949 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:33:58.233227 14027 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 2.
I20260806 20:33:58.234220 14070 leader_election.cc:302] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:33:58.234555 13949 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 2.
I20260806 20:33:58.235082 14239 raft_consensus.cc:2781] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:33:58.235563 14239 raft_consensus.cc:686] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:33:58.236377 14239 consensus_queue.cc:237] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:33:58.244422 13846 catalog_manager.cc:5495] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 1 to 2, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 2 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:33:58.582505 14027 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:58.583624 14239 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:58.592782 13949 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:33:58.594823 14239 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:33:58.711687 13949 raft_consensus.cc:1261] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:58.712888 14239 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:58.721549 14027 raft_consensus.cc:1261] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:33:58.723127 14260 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:33:58.974994 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:58.980304 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:58.980695 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 20:33:59.429448 14050 debug-util.cc:398] Leaking SignalData structure 0x7b08001044a0 after lost signal to thread 13285
W20260806 20:33:59.430409 14050 debug-util.cc:398] Leaking SignalData structure 0x7b0800104b80 after lost signal to thread 13880
W20260806 20:33:59.431334 14050 debug-util.cc:398] Leaking SignalData structure 0x7b0800203500 after lost signal to thread 13977
W20260806 20:33:59.432176 14050 debug-util.cc:398] Leaking SignalData structure 0x7b08001f5d60 after lost signal to thread 14053
W20260806 20:33:59.433104 14050 debug-util.cc:398] Leaking SignalData structure 0x7b08001e11c0 after lost signal to thread 14130
I20260806 20:33:59.981683 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:33:59.990631 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:33:59.991228 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:00.992079 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:00.998703 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:00.999375 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:02.000437 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:02.006552 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:02.007225 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:03.008366 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:03.014132 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:03.014606 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:04.015417 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:04.020831 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:04.021282 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:05.022090 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:05.028213 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:05.028741 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:06.029623 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:06.035110 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:06.035481 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:07.036201 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:07.041540 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:07.041931 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:08.072290 13282 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260806 20:34:08.076911 14106 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:59636
I20260806 20:34:08.077858 14106 raft_consensus.cc:595] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:34:08.082361 14106 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:59636
I20260806 20:34:08.082840 14106 raft_consensus.cc:595] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:34:08.094209 14028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:44500
I20260806 20:34:08.094722 14028 raft_consensus.cc:595] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:34:08.099117 14028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:44500
I20260806 20:34:08.099644 14028 raft_consensus.cc:595] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:34:08.103971 14028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
mode: GRACEFUL
new_leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:44500
I20260806 20:34:08.104359 14028 raft_consensus.cc:595] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Received request to transfer leadership to TS 8dfa79a66fe34b4ebb92589c7e06caf0
I20260806 20:34:08.122520 14273 raft_consensus.cc:979] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.123208 14278 raft_consensus.cc:1067] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.124941 13951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50792
I20260806 20:34:08.125461 13951 raft_consensus.cc:482] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:08.125923 13951 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.131796 13951 raft_consensus.cc:504] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.133791 13951 leader_election.cc:288] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 3 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:34:08.134804 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:34:08.134588 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:34:08.135407 14106 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.135468 14028 raft_consensus.cc:3032] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:34:08.135833 14028 raft_consensus.cc:729] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:34:08.136533 14028 consensus_queue.cc:260] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.137506 14028 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.139918 14106 raft_consensus.cc:2454] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 3.
I20260806 20:34:08.141072 13914 leader_election.cc:302] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:34:08.141884 14244 raft_consensus.cc:2781] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:34:08.141971 14028 raft_consensus.cc:2454] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 3.
I20260806 20:34:08.143077 14244 raft_consensus.cc:686] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:08.143967 14244 consensus_queue.cc:237] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.151026 13844 catalog_manager.cc:5495] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 2 to 3, leader changed from b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 3 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:08.183089 14260 raft_consensus.cc:979] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.183615 14287 raft_consensus.cc:1067] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.184984 13951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50794
I20260806 20:34:08.185479 13951 raft_consensus.cc:482] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:08.185801 13951 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.190605 13951 raft_consensus.cc:504] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.192129 13951 leader_election.cc:288] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 3 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:34:08.193012 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:34:08.193122 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:34:08.193562 14028 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.193630 14106 raft_consensus.cc:3032] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:34:08.193998 14106 raft_consensus.cc:729] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:34:08.194581 14106 consensus_queue.cc:260] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.195518 14106 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.197927 14028 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 3.
I20260806 20:34:08.198869 13915 leader_election.cc:302] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:34:08.199527 14244 raft_consensus.cc:2781] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:34:08.199937 14244 raft_consensus.cc:686] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:08.200433 14106 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 3.
I20260806 20:34:08.200801 14244 consensus_queue.cc:237] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.208740 13844 catalog_manager.cc:5495] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 2 to 3, leader changed from a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 3 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:08.242795 14278 raft_consensus.cc:979] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.243387 14273 raft_consensus.cc:1067] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.245225 13951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50792
I20260806 20:34:08.245894 13951 raft_consensus.cc:482] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:08.246318 13951 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.251724 13951 raft_consensus.cc:504] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.253351 13951 leader_election.cc:288] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 3 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:34:08.254189 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:34:08.254357 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4283315fbead43cbad6478f9d5233a97" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:34:08.254756 14028 raft_consensus.cc:3032] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:34:08.254832 14106 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.255128 14028 raft_consensus.cc:729] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:34:08.255860 14028 consensus_queue.cc:260] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.256951 14028 raft_consensus.cc:3037] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:08.260159 14106 raft_consensus.cc:2454] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 3.
I20260806 20:34:08.261122 13914 leader_election.cc:302] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:34:08.261865 14244 raft_consensus.cc:2781] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:34:08.261868 14028 raft_consensus.cc:2454] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 3.
I20260806 20:34:08.262437 14244 raft_consensus.cc:686] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:08.263226 14244 consensus_queue.cc:237] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.271224 13844 catalog_manager.cc:5495] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 2 to 3, leader changed from b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 3 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:08.418066 14278 raft_consensus.cc:979] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.418537 14273 raft_consensus.cc:1067] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.419963 13951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50792
I20260806 20:34:08.420392 13951 raft_consensus.cc:482] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:08.420670 13951 raft_consensus.cc:3037] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:08.425767 13951 raft_consensus.cc:504] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.427083 13951 leader_election.cc:288] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:34:08.427855 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:34:08.428085 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703d6b0c741743debb9005ad6b0606aa" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:34:08.428488 14028 raft_consensus.cc:3032] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:08.428579 14106 raft_consensus.cc:3037] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:08.428813 14028 raft_consensus.cc:729] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:34:08.429549 14028 consensus_queue.cc:260] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.430807 14028 raft_consensus.cc:3037] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:08.434932 14106 raft_consensus.cc:2454] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:34:08.436038 13914 leader_election.cc:302] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, a6a8d885dee24fe6b9541df33ffc43b5; no voters:
I20260806 20:34:08.436810 14244 raft_consensus.cc:2781] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:08.437208 14244 raft_consensus.cc:686] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:08.437305 14028 raft_consensus.cc:2454] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 2.
I20260806 20:34:08.437932 14244 consensus_queue.cc:237] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:08.444880 13844 catalog_manager.cc:5495] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 1 to 2, leader changed from b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:08.587442 14028 raft_consensus.cc:1261] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:08.588618 14244 consensus_queue.cc:1035] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:08.589680 14260 raft_consensus.cc:979] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5: : Instructing follower 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.590117 14287 raft_consensus.cc:1067] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 LEADER]: Signalling peer 8dfa79a66fe34b4ebb92589c7e06caf0 to start an election
I20260806 20:34:08.591450 13951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
from {username='slave'} at 127.0.0.1:50794
I20260806 20:34:08.591888 13951 raft_consensus.cc:482] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:08.592481 13951 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:08.599166 13951 raft_consensus.cc:504] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:34:08.602171 13951 leader_election.cc:288] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 4 election: Requested vote from peers a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:34:08.604563 14106 raft_consensus.cc:1261] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:08.604888 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:34:08.605491 14103 raft_consensus.cc:3032] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:34:08.605821 14103 raft_consensus.cc:729] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:34:08.606146 14027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:34:08.606726 14027 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:08.606667 14103 consensus_queue.cc:260] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:34:08.609186 14280 consensus_queue.cc:1035] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:08.608057 14103 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:08.615100 14027 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 4.
I20260806 20:34:08.616282 13915 leader_election.cc:302] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfa79a66fe34b4ebb92589c7e06caf0, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:34:08.617080 14280 raft_consensus.cc:2781] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:34:08.617527 14280 raft_consensus.cc:686] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 LEADER]: Becoming Leader. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:08.618453 14280 consensus_queue.cc:237] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:34:08.620631 14103 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfa79a66fe34b4ebb92589c7e06caf0 in term 4.
I20260806 20:34:08.626874 13844 catalog_manager.cc:5495] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 reported cstate change: term changed from 3 to 4, leader changed from a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131) to 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129). New cstate: current_term: 4 leader_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:08.674484 14106 raft_consensus.cc:1261] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:08.675794 14280 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:08.685948 14027 raft_consensus.cc:1261] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:08.687196 14280 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:08.717795 14106 raft_consensus.cc:1261] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:08.719182 14294 consensus_queue.cc:1035] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:08.720304 14027 raft_consensus.cc:1261] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:08.721455 14294 consensus_queue.cc:1035] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:08.893374 14027 raft_consensus.cc:1261] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:08.894435 14294 consensus_queue.cc:1035] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:08.902513 14106 raft_consensus.cc:1261] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:08.904102 14244 consensus_queue.cc:1035] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:09.058820 14106 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:34:09.059916 14294 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:09.068379 14027 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Refusing update from remote peer 8dfa79a66fe34b4ebb92589c7e06caf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:34:09.069845 14294 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:11.180927 13282 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260806 20:34:11.181579 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:11.188155 13951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:46726
I20260806 20:34:11.188678 13951 raft_consensus.cc:595] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Received request to transfer leadership to TS a6a8d885dee24fe6b9541df33ffc43b5
I20260806 20:34:11.190788 13951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:46726
I20260806 20:34:11.191190 13951 raft_consensus.cc:595] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 LEADER]: Received request to transfer leadership to TS a6a8d885dee24fe6b9541df33ffc43b5
I20260806 20:34:11.193104 13951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:46726
I20260806 20:34:11.193501 13951 raft_consensus.cc:595] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Received request to transfer leadership to TS b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:34:11.195345 13951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29"
dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
mode: GRACEFUL
new_leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:46726
I20260806 20:34:11.195748 13951 raft_consensus.cc:595] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Received request to transfer leadership to TS b4bdb3cf462e488bb99dca8f2ddf0148
I20260806 20:34:11.196555 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 20:34:11.196911 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:11.392686 14307 raft_consensus.cc:979] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:34:11.393244 14306 raft_consensus.cc:1067] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Signalling peer b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:34:11.394558 14028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:44472
I20260806 20:34:11.394946 14028 raft_consensus.cc:482] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:11.395159 14028 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:11.399657 14028 raft_consensus.cc:504] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.401515 14028 leader_election.cc:288] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 4 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:34:11.402393 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:34:11.402942 13951 raft_consensus.cc:3032] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:34:11.403039 14104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d04a9ade94dc8a276327180865e29" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:34:11.403231 13951 raft_consensus.cc:729] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:11.403587 14104 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:11.403867 13951 consensus_queue.cc:260] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.404882 13951 raft_consensus.cc:3037] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:11.408601 14104 raft_consensus.cc:2454] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 4.
I20260806 20:34:11.409089 13951 raft_consensus.cc:2454] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 4.
I20260806 20:34:11.409827 13993 leader_election.cc:302] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:34:11.410573 14273 raft_consensus.cc:2781] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:34:11.411360 14273 raft_consensus.cc:686] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 LEADER]: Becoming Leader. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:34:11.412000 14273 consensus_queue.cc:237] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.421092 13844 catalog_manager.cc:5495] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 reported cstate change: term changed from 3 to 4, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130). New cstate: current_term: 4 leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:11.480620 14307 raft_consensus.cc:979] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:34:11.481079 14306 raft_consensus.cc:1067] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 LEADER]: Signalling peer a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:34:11.482568 14104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:59628
I20260806 20:34:11.483052 14104 raft_consensus.cc:482] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:11.483366 14104 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:34:11.487818 14104 raft_consensus.cc:504] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:34:11.489840 14104 leader_election.cc:288] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 5 election: Requested vote from peers b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575), 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891)
I20260806 20:34:11.490653 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:34:11.490927 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e62f8f05af4555ae03bff83152c07d" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:34:11.491242 14028 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:34:11.491496 13951 raft_consensus.cc:3032] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:34:11.491771 13951 raft_consensus.cc:729] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:11.492409 13951 consensus_queue.cc:260] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:34:11.493618 13951 raft_consensus.cc:3037] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:34:11.495791 14028 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 5.
I20260806 20:34:11.496815 14070 leader_election.cc:302] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:34:11.497468 14287 raft_consensus.cc:2781] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:34:11.498606 14287 raft_consensus.cc:686] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 5 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:34:11.499148 13951 raft_consensus.cc:2454] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 5.
I20260806 20:34:11.499397 14287 consensus_queue.cc:237] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } }
I20260806 20:34:11.506119 13844 catalog_manager.cc:5495] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 4 to 5, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 5 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:11.573427 14307 raft_consensus.cc:979] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:34:11.573877 14306 raft_consensus.cc:1067] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Signalling peer b4bdb3cf462e488bb99dca8f2ddf0148 to start an election
I20260806 20:34:11.575188 14028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1"
dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
from {username='slave'} at 127.0.0.1:44472
I20260806 20:34:11.575629 14028 raft_consensus.cc:482] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:11.575909 14028 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:11.580013 14028 raft_consensus.cc:504] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.581328 14028 leader_election.cc:288] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 4 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131:42121)
I20260806 20:34:11.582288 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:34:11.582389 14104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9221c76981a948cb9aa3ccffbb6843a1" candidate_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
I20260806 20:34:11.582842 13951 raft_consensus.cc:3032] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:34:11.582909 14104 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:11.583246 13951 raft_consensus.cc:729] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:11.584053 13951 consensus_queue.cc:260] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.585342 13951 raft_consensus.cc:3037] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:34:11.590169 14104 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 4.
I20260806 20:34:11.591285 13993 leader_election.cc:302] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:34:11.591724 13951 raft_consensus.cc:2454] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bdb3cf462e488bb99dca8f2ddf0148 in term 4.
I20260806 20:34:11.591967 14273 raft_consensus.cc:2781] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:34:11.592540 14273 raft_consensus.cc:686] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 LEADER]: Becoming Leader. State: Replica: b4bdb3cf462e488bb99dca8f2ddf0148, State: Running, Role: LEADER
I20260806 20:34:11.593501 14273 consensus_queue.cc:237] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.600252 13844 catalog_manager.cc:5495] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 reported cstate change: term changed from 3 to 4, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130). New cstate: current_term: 4 leader_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:11.621904 14307 raft_consensus.cc:979] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0: : Instructing follower a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:34:11.622464 14306 raft_consensus.cc:1067] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Signalling peer a6a8d885dee24fe6b9541df33ffc43b5 to start an election
I20260806 20:34:11.623746 14104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2"
dest_uuid: "a6a8d885dee24fe6b9541df33ffc43b5"
from {username='slave'} at 127.0.0.1:59628
I20260806 20:34:11.624210 14104 raft_consensus.cc:482] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:11.624527 14104 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:11.629076 14104 raft_consensus.cc:504] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.631419 14104 leader_election.cc:288] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 3 election: Requested vote from peers 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891), b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575)
I20260806 20:34:11.631824 14028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148"
I20260806 20:34:11.632401 14028 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:11.632498 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e70b403d6754349ab3616a0e32cc2a2" candidate_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0"
I20260806 20:34:11.633306 13951 raft_consensus.cc:3032] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:34:11.633682 13951 raft_consensus.cc:729] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8dfa79a66fe34b4ebb92589c7e06caf0, State: Running, Role: LEADER
I20260806 20:34:11.634407 13951 consensus_queue.cc:260] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.635242 13951 raft_consensus.cc:3037] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:11.637305 14028 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 3.
I20260806 20:34:11.638137 14070 leader_election.cc:302] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a8d885dee24fe6b9541df33ffc43b5, b4bdb3cf462e488bb99dca8f2ddf0148; no voters:
I20260806 20:34:11.638801 14287 raft_consensus.cc:2781] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:34:11.639360 14287 raft_consensus.cc:686] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 LEADER]: Becoming Leader. State: Replica: a6a8d885dee24fe6b9541df33ffc43b5, State: Running, Role: LEADER
I20260806 20:34:11.640151 13951 raft_consensus.cc:2454] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a8d885dee24fe6b9541df33ffc43b5 in term 3.
I20260806 20:34:11.640249 14287 consensus_queue.cc:237] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } }
I20260806 20:34:11.647339 13844 catalog_manager.cc:5495] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 reported cstate change: term changed from 2 to 3, leader changed from 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129) to a6a8d885dee24fe6b9541df33ffc43b5 (127.12.248.131). New cstate: current_term: 3 leader_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:11.822032 13951 raft_consensus.cc:1261] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:34:11.823364 14273 consensus_queue.cc:1035] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:11.834213 14104 raft_consensus.cc:1261] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:34:11.835579 14321 consensus_queue.cc:1035] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:11.949788 13951 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 5 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:34:11.951025 14287 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:34:11.959740 14027 raft_consensus.cc:1261] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 5 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:34:11.961210 14314 consensus_queue.cc:1035] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260806 20:34:12.019690 13972 debug-util.cc:398] Leaking SignalData structure 0x7b080010c8a0 after lost signal to thread 13285
W20260806 20:34:12.020645 13972 debug-util.cc:398] Leaking SignalData structure 0x7b08001e6f20 after lost signal to thread 13880
W20260806 20:34:12.021528 13972 debug-util.cc:398] Leaking SignalData structure 0x7b08001fedc0 after lost signal to thread 13977
W20260806 20:34:12.022284 13972 debug-util.cc:398] Leaking SignalData structure 0x7b08001caaa0 after lost signal to thread 14053
W20260806 20:34:12.023020 13972 debug-util.cc:398] Leaking SignalData structure 0x7b08000e0aa0 after lost signal to thread 14130
I20260806 20:34:12.061422 13951 raft_consensus.cc:1261] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:34:12.062537 14321 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:12.071326 14104 raft_consensus.cc:1261] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Refusing update from remote peer b4bdb3cf462e488bb99dca8f2ddf0148: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:34:12.072975 14308 consensus_queue.cc:1035] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a8d885dee24fe6b9541df33ffc43b5" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:12.148799 14027 raft_consensus.cc:1261] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:12.150152 14314 consensus_queue.cc:1035] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bdb3cf462e488bb99dca8f2ddf0148" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:12.162070 13951 raft_consensus.cc:1261] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Refusing update from remote peer a6a8d885dee24fe6b9541df33ffc43b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:12.164247 14287 consensus_queue.cc:1035] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfa79a66fe34b4ebb92589c7e06caf0" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:12.198282 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:12.215405 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:12.216089 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:13.216948 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:13.222457 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:13.222859 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:14.223623 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:14.229193 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:14.229597 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:15.230358 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:15.236255 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:15.236688 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:16.237720 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:16.247022 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:16.247576 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:17.248486 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:17.253872 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:17.254220 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:18.254963 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:18.259958 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:18.260295 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:19.261025 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:19.266093 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:19.266599 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:20.267414 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:20.272571 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:34:20.272992 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:21.301859 13282 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260806 20:34:21.302361 13282 tablet_server.cc:178] TabletServer@127.12.248.129:0 shutting down...
W20260806 20:34:21.329192 14068 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20260806 20:34:21.332172 13282 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:34:21.333000 13282 tablet_replica.cc:330] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
W20260806 20:34:21.333344 14068 consensus_peers.cc:487] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 -> Peer 8dfa79a66fe34b4ebb92589c7e06caf0 (127.12.248.129:45891): Couldn't send request to peer 8dfa79a66fe34b4ebb92589c7e06caf0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:21.333966 13282 raft_consensus.cc:2229] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.334990 13282 raft_consensus.cc:2258] T 703d6b0c741743debb9005ad6b0606aa P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.337069 13282 tablet_replica.cc:330] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.337633 13282 raft_consensus.cc:2229] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.338058 13282 raft_consensus.cc:2258] T 9a0d04a9ade94dc8a276327180865e29 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.339898 13282 tablet_replica.cc:330] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.340382 13282 raft_consensus.cc:2229] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.341190 13282 raft_consensus.cc:2258] T fdc43286cc67445f9d0462d09d2f1c3f P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.343001 13282 tablet_replica.cc:330] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.343611 13282 raft_consensus.cc:2229] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.344038 13282 raft_consensus.cc:2258] T 45e62f8f05af4555ae03bff83152c07d P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.345731 13282 tablet_replica.cc:330] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.346218 13282 raft_consensus.cc:2229] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.346675 13282 raft_consensus.cc:2258] T be40a4b3cc7744c99b0a1ecd3f2d60ff P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.348585 13282 tablet_replica.cc:330] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.349092 13282 raft_consensus.cc:2229] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.349529 13282 raft_consensus.cc:2258] T 2e70b403d6754349ab3616a0e32cc2a2 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.351204 13282 tablet_replica.cc:330] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.351660 13282 raft_consensus.cc:2229] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.352105 13282 raft_consensus.cc:2258] T 9221c76981a948cb9aa3ccffbb6843a1 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.353817 13282 tablet_replica.cc:330] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.354274 13282 raft_consensus.cc:2229] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.354666 13282 raft_consensus.cc:2258] T 09da2a02cc3e4ae092b4757a1ebd5b53 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.356141 13282 tablet_replica.cc:330] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.356673 13282 raft_consensus.cc:2229] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.357410 13282 raft_consensus.cc:2258] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.359010 13282 tablet_replica.cc:330] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0: stopping tablet replica
I20260806 20:34:21.359436 13282 raft_consensus.cc:2229] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.360193 13282 raft_consensus.cc:2258] T 4283315fbead43cbad6478f9d5233a97 P 8dfa79a66fe34b4ebb92589c7e06caf0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.405298 13282 tablet_server.cc:194] TabletServer@127.12.248.129:0 shutdown complete.
I20260806 20:34:21.425036 13282 tablet_server.cc:178] TabletServer@127.12.248.130:0 shutting down...
I20260806 20:34:21.454702 13282 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:34:21.455253 13282 tablet_replica.cc:330] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.455859 13282 raft_consensus.cc:2229] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.456781 13282 raft_consensus.cc:2258] T 09da2a02cc3e4ae092b4757a1ebd5b53 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.458644 13282 tablet_replica.cc:330] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.459262 13282 raft_consensus.cc:2229] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.459972 13282 raft_consensus.cc:2258] T 9221c76981a948cb9aa3ccffbb6843a1 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.461787 13282 tablet_replica.cc:330] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.462415 13282 raft_consensus.cc:2229] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.462916 13282 raft_consensus.cc:2258] T 703d6b0c741743debb9005ad6b0606aa P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.465335 13282 tablet_replica.cc:330] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.466029 13282 raft_consensus.cc:2229] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260806 20:34:21.466324 14070 consensus_peers.cc:487] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 -> Peer b4bdb3cf462e488bb99dca8f2ddf0148 (127.12.248.130:38575): Couldn't send request to peer b4bdb3cf462e488bb99dca8f2ddf0148. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:21.466557 13282 raft_consensus.cc:2258] T 2e70b403d6754349ab3616a0e32cc2a2 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.468365 13282 tablet_replica.cc:330] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.468914 13282 raft_consensus.cc:2229] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.469369 13282 raft_consensus.cc:2258] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.470983 13282 tablet_replica.cc:330] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.471448 13282 raft_consensus.cc:2229] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.471854 13282 raft_consensus.cc:2258] T 4283315fbead43cbad6478f9d5233a97 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.473476 13282 tablet_replica.cc:330] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.473958 13282 raft_consensus.cc:2229] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.474344 13282 raft_consensus.cc:2258] T fdc43286cc67445f9d0462d09d2f1c3f P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.475868 13282 tablet_replica.cc:330] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.476341 13282 raft_consensus.cc:2229] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.477133 13282 raft_consensus.cc:2258] T 9a0d04a9ade94dc8a276327180865e29 P b4bdb3cf462e488bb99dca8f2ddf0148 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.478703 13282 tablet_replica.cc:330] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.479151 13282 raft_consensus.cc:2229] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.479540 13282 raft_consensus.cc:2258] T be40a4b3cc7744c99b0a1ecd3f2d60ff P b4bdb3cf462e488bb99dca8f2ddf0148 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.481101 13282 tablet_replica.cc:330] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148: stopping tablet replica
I20260806 20:34:21.481510 13282 raft_consensus.cc:2229] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.481954 13282 raft_consensus.cc:2258] T 45e62f8f05af4555ae03bff83152c07d P b4bdb3cf462e488bb99dca8f2ddf0148 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.526365 13282 tablet_server.cc:194] TabletServer@127.12.248.130:0 shutdown complete.
I20260806 20:34:21.545112 13282 tablet_server.cc:178] TabletServer@127.12.248.131:0 shutting down...
I20260806 20:34:21.576696 13282 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:34:21.577322 13282 tablet_replica.cc:330] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.577963 13282 raft_consensus.cc:2229] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.578478 13282 raft_consensus.cc:2258] T 703d6b0c741743debb9005ad6b0606aa P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.580226 13282 tablet_replica.cc:330] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.580832 13282 raft_consensus.cc:2229] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.581669 13282 raft_consensus.cc:2258] T be40a4b3cc7744c99b0a1ecd3f2d60ff P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.583280 13282 tablet_replica.cc:330] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.583777 13282 raft_consensus.cc:2229] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 5 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.584509 13282 raft_consensus.cc:2258] T 45e62f8f05af4555ae03bff83152c07d P a6a8d885dee24fe6b9541df33ffc43b5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.586086 13282 tablet_replica.cc:330] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.586550 13282 raft_consensus.cc:2229] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.586973 13282 raft_consensus.cc:2258] T 9221c76981a948cb9aa3ccffbb6843a1 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.588586 13282 tablet_replica.cc:330] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.589051 13282 raft_consensus.cc:2229] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.589480 13282 raft_consensus.cc:2258] T 09da2a02cc3e4ae092b4757a1ebd5b53 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.591034 13282 tablet_replica.cc:330] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.591485 13282 raft_consensus.cc:2229] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.591876 13282 raft_consensus.cc:2258] T 4283315fbead43cbad6478f9d5233a97 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.593406 13282 tablet_replica.cc:330] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.593865 13282 raft_consensus.cc:2229] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.594277 13282 raft_consensus.cc:2258] T 938f51cb2ffe41f7a9cfe6f99c5f0634 P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.595747 13282 tablet_replica.cc:330] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.596195 13282 raft_consensus.cc:2229] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.596953 13282 raft_consensus.cc:2258] T 2e70b403d6754349ab3616a0e32cc2a2 P a6a8d885dee24fe6b9541df33ffc43b5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.598527 13282 tablet_replica.cc:330] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.598989 13282 raft_consensus.cc:2229] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.599418 13282 raft_consensus.cc:2258] T fdc43286cc67445f9d0462d09d2f1c3f P a6a8d885dee24fe6b9541df33ffc43b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.601032 13282 tablet_replica.cc:330] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5: stopping tablet replica
I20260806 20:34:21.601455 13282 raft_consensus.cc:2229] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:21.601850 13282 raft_consensus.cc:2258] T 9a0d04a9ade94dc8a276327180865e29 P a6a8d885dee24fe6b9541df33ffc43b5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.630798 13282 tablet_server.cc:194] TabletServer@127.12.248.131:0 shutdown complete.
I20260806 20:34:21.650656 13282 master.cc:537] Master@127.12.248.190:36203 shutting down...
I20260806 20:34:21.683362 13282 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:34:21.684032 13282 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:21.684418 13282 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78141565cd3f4653a9e0c21f5d20d7fd: stopping tablet replica
I20260806 20:34:21.703063 13282 master.cc:552] Master@127.12.248.190:36203 shutdown complete.
8dfa79a66fe34b4ebb92589c7e06caf0 5
a6a8d885dee24fe6b9541df33ffc43b5 1
b4bdb3cf462e488bb99dca8f2ddf0148 4
8dfa79a66fe34b4ebb92589c7e06caf0 3
a6a8d885dee24fe6b9541df33ffc43b5 3
b4bdb3cf462e488bb99dca8f2ddf0148 4
8dfa79a66fe34b4ebb92589c7e06caf0 8
a6a8d885dee24fe6b9541df33ffc43b5 1
b4bdb3cf462e488bb99dca8f2ddf0148 1
8dfa79a66fe34b4ebb92589c7e06caf0 4
a6a8d885dee24fe6b9541df33ffc43b5 3
b4bdb3cf462e488bb99dca8f2ddf0148 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28556 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260806 20:34:21.745353 13282 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.248.190:39439
I20260806 20:34:21.746448 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:34:21.752154 14347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:21.752326 14346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:21.753391 14349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:34:21.754539 13282 server_base.cc:1034] running on GCE node
I20260806 20:34:21.755592 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:34:21.755808 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:34:21.755966 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048461755951 us; error 0 us; skew 500 ppm
I20260806 20:34:21.756548 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:34:21.764907 13282 webserver.cc:460] Webserver started at http://127.12.248.190:34777/ using document root <none> and password file <none>
I20260806 20:34:21.765424 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:34:21.765630 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:34:21.765926 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:34:21.767165 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/master-0-root/instance:
uuid: "5c7ed2f344954f8c90dbc753df8f30f7"
format_stamp: "Formatted at 2026-08-06 20:34:21 on dist-test-slave-10pc"
I20260806 20:34:21.771596 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 20:34:21.774765 14354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:21.775429 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 20:34:21.775702 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/master-0-root
uuid: "5c7ed2f344954f8c90dbc753df8f30f7"
format_stamp: "Formatted at 2026-08-06 20:34:21 on dist-test-slave-10pc"
I20260806 20:34:21.775965 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:34:21.799639 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:34:21.800987 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:34:21.839468 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.190:39439
I20260806 20:34:21.839576 14407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.190:39439 every 8 connection(s)
I20260806 20:34:21.843217 14408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:21.852885 14408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Bootstrap starting.
I20260806 20:34:21.857183 14408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:21.861526 14408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: No bootstrap required, opened a new log
I20260806 20:34:21.863526 14408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" member_type: VOTER }
I20260806 20:34:21.864042 14408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:21.864338 14408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7ed2f344954f8c90dbc753df8f30f7, State: Initialized, Role: FOLLOWER
I20260806 20:34:21.865028 14408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" member_type: VOTER }
I20260806 20:34:21.865476 14408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:34:21.865724 14408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:34:21.866016 14408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:21.869871 14408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" member_type: VOTER }
I20260806 20:34:21.870359 14408 leader_election.cc:302] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c7ed2f344954f8c90dbc753df8f30f7; no voters:
I20260806 20:34:21.871497 14408 leader_election.cc:288] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:34:21.871913 14411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:21.873286 14411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5c7ed2f344954f8c90dbc753df8f30f7, State: Running, Role: LEADER
I20260806 20:34:21.873956 14411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" member_type: VOTER }
I20260806 20:34:21.874531 14408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:34:21.879158 14412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" member_type: VOTER } }
I20260806 20:34:21.879933 14413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c7ed2f344954f8c90dbc753df8f30f7. Latest consensus state: current_term: 1 leader_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7ed2f344954f8c90dbc753df8f30f7" member_type: VOTER } }
I20260806 20:34:21.879992 14412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [sys.catalog]: This master's current role is: LEADER
I20260806 20:34:21.880487 14413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7 [sys.catalog]: This master's current role is: LEADER
I20260806 20:34:21.882400 14419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:34:21.887537 14419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:34:21.890113 13282 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 20:34:21.896628 14419 catalog_manager.cc:1347] Generated new cluster ID: 67ad796e8aa7438d996ccec49531e31f
I20260806 20:34:21.896872 14419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:34:21.934638 14419 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:34:21.936342 14419 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:34:21.962466 14419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Generated new TSK 0
I20260806 20:34:21.963160 14419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:34:22.020865 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:34:22.026978 14431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:22.028334 14432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:34:22.030390 13282 server_base.cc:1034] running on GCE node
W20260806 20:34:22.030352 14434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:34:22.031695 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:34:22.031917 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:34:22.032099 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048462032083 us; error 0 us; skew 500 ppm
I20260806 20:34:22.032702 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:34:22.035009 13282 webserver.cc:460] Webserver started at http://127.12.248.129:41567/ using document root <none> and password file <none>
I20260806 20:34:22.035516 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:34:22.035725 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:34:22.035990 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:34:22.037138 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/instance:
uuid: "92828dd6dbe74f29a04d4e702b502621"
format_stamp: "Formatted at 2026-08-06 20:34:22 on dist-test-slave-10pc"
I20260806 20:34:22.041579 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:34:22.044718 14440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.045646 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 20:34:22.045894 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root
uuid: "92828dd6dbe74f29a04d4e702b502621"
format_stamp: "Formatted at 2026-08-06 20:34:22 on dist-test-slave-10pc"
I20260806 20:34:22.046139 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:34:22.059101 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:34:22.060339 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:34:22.061791 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:34:22.064023 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:34:22.064193 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.064488 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:34:22.064651 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 20:34:22.105504 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.129:44789
I20260806 20:34:22.105599 14503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.129:44789 every 8 connection(s)
I20260806 20:34:22.110006 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:34:22.117863 14509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:22.119576 14510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:34:22.122273 13282 server_base.cc:1034] running on GCE node
W20260806 20:34:22.122721 14512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:34:22.123947 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:34:22.124207 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:34:22.124389 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048462124357 us; error 0 us; skew 500 ppm
I20260806 20:34:22.124976 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:34:22.126782 14504 heartbeater.cc:347] Connected to a master server at 127.12.248.190:39439
I20260806 20:34:22.127171 14504 heartbeater.cc:464] Registering TS with master...
I20260806 20:34:22.127725 13282 webserver.cc:460] Webserver started at http://127.12.248.130:39883/ using document root <none> and password file <none>
I20260806 20:34:22.128054 14504 heartbeater.cc:511] Master 127.12.248.190:39439 requested a full tablet report, sending...
I20260806 20:34:22.128211 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:34:22.128511 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:34:22.128786 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:34:22.130275 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/instance:
uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
format_stamp: "Formatted at 2026-08-06 20:34:22 on dist-test-slave-10pc"
I20260806 20:34:22.130311 14372 ts_manager.cc:194] Registered new tserver with Master: 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:22.133113 14372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45942
I20260806 20:34:22.136245 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260806 20:34:22.139465 14517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.140210 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 20:34:22.140529 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root
uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
format_stamp: "Formatted at 2026-08-06 20:34:22 on dist-test-slave-10pc"
I20260806 20:34:22.140797 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:34:22.184778 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:34:22.186002 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:34:22.187526 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:34:22.189720 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:34:22.189930 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.190171 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:34:22.190332 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.256646 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.130:39545
I20260806 20:34:22.256783 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.130:39545 every 8 connection(s)
I20260806 20:34:22.263455 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:34:22.273557 14587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:22.273018 14586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:22.280099 14589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:34:22.280385 13282 server_base.cc:1034] running on GCE node
I20260806 20:34:22.281708 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:34:22.282027 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:34:22.282238 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048462282222 us; error 0 us; skew 500 ppm
I20260806 20:34:22.282433 14582 heartbeater.cc:347] Connected to a master server at 127.12.248.190:39439
I20260806 20:34:22.282840 14582 heartbeater.cc:464] Registering TS with master...
I20260806 20:34:22.283000 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:34:22.283842 14582 heartbeater.cc:511] Master 127.12.248.190:39439 requested a full tablet report, sending...
I20260806 20:34:22.286844 13282 webserver.cc:460] Webserver started at http://127.12.248.131:36361/ using document root <none> and password file <none>
I20260806 20:34:22.286979 14372 ts_manager.cc:194] Registered new tserver with Master: 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:22.287650 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:34:22.287976 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:34:22.288386 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:34:22.289700 14372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45954
I20260806 20:34:22.290611 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/instance:
uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
format_stamp: "Formatted at 2026-08-06 20:34:22 on dist-test-slave-10pc"
I20260806 20:34:22.296240 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:34:22.299652 14594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.300423 13282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 20:34:22.300745 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root
uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
format_stamp: "Formatted at 2026-08-06 20:34:22 on dist-test-slave-10pc"
I20260806 20:34:22.300980 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:34:22.315380 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:34:22.316701 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:34:22.318233 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:34:22.321350 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:34:22.321637 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.321908 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:34:22.322093 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:22.367516 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.131:45209
I20260806 20:34:22.367704 14657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.131:45209 every 8 connection(s)
I20260806 20:34:22.384272 14658 heartbeater.cc:347] Connected to a master server at 127.12.248.190:39439
I20260806 20:34:22.384773 14658 heartbeater.cc:464] Registering TS with master...
I20260806 20:34:22.385491 14658 heartbeater.cc:511] Master 127.12.248.190:39439 requested a full tablet report, sending...
I20260806 20:34:22.387360 14372 ts_manager.cc:194] Registered new tserver with Master: df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:22.387969 13282 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017194229s
I20260806 20:34:22.389556 14372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45970
I20260806 20:34:22.391393 13282 test_util.cc:268] Using random seed: -802748339
I20260806 20:34:22.416886 14372 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:34:22.423501 14372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:34:22.648241 14545 tablet_service.cc:1467] Processing CreateTablet for tablet 8844659c5dd24572bab900a68fc14df6 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 20:34:22.650861 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 8844659c5dd24572bab900a68fc14df6 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 20:34:22.651122 14537 tablet_service.cc:1467] Processing CreateTablet for tablet 5b527f3bc7bb43f88f1b17c445c3c0d0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 20:34:22.653534 14463 tablet_service.cc:1467] Processing CreateTablet for tablet f38960b1dd9245a8ba85fea601b09e94 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 20:34:22.653710 14467 tablet_service.cc:1467] Processing CreateTablet for tablet fcae2b574a134bb781ba6c1a424f84a4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 20:34:22.654821 14464 tablet_service.cc:1467] Processing CreateTablet for tablet 3044f79ab3f74f788d8a263e141674a8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 20:34:22.655019 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 8986697653e048048cd32eb27e4ae3d4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 20:34:22.655627 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8844659c5dd24572bab900a68fc14df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.656288 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae2b574a134bb781ba6c1a424f84a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.657359 14465 tablet_service.cc:1467] Processing CreateTablet for tablet fe354ed675014a3192fc7fdc9937479c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 20:34:22.657483 14542 tablet_service.cc:1467] Processing CreateTablet for tablet fe354ed675014a3192fc7fdc9937479c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 20:34:22.658011 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38960b1dd9245a8ba85fea601b09e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.658684 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe354ed675014a3192fc7fdc9937479c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.658018 14462 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 20:34:22.659901 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 3044f79ab3f74f788d8a263e141674a8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 20:34:22.661922 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 8986697653e048048cd32eb27e4ae3d4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 20:34:22.662508 14538 tablet_service.cc:1467] Processing CreateTablet for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 20:34:22.660898 14461 tablet_service.cc:1467] Processing CreateTablet for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 20:34:22.664098 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.664913 14536 tablet_service.cc:1467] Processing CreateTablet for tablet 4248ec88bd4b44e990938a74fe3c4fdc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 20:34:22.667287 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 20:34:22.669276 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.669590 14540 tablet_service.cc:1467] Processing CreateTablet for tablet f38960b1dd9245a8ba85fea601b09e94 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 20:34:22.670809 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8844659c5dd24572bab900a68fc14df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.673489 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8986697653e048048cd32eb27e4ae3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.676095 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3044f79ab3f74f788d8a263e141674a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.676793 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b527f3bc7bb43f88f1b17c445c3c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.680538 14622 tablet_service.cc:1467] Processing CreateTablet for tablet 8844659c5dd24572bab900a68fc14df6 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 20:34:22.681851 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8844659c5dd24572bab900a68fc14df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.684769 14621 tablet_service.cc:1467] Processing CreateTablet for tablet fcae2b574a134bb781ba6c1a424f84a4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 20:34:22.686174 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae2b574a134bb781ba6c1a424f84a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.689644 14620 tablet_service.cc:1467] Processing CreateTablet for tablet 8986697653e048048cd32eb27e4ae3d4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 20:34:22.690986 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8986697653e048048cd32eb27e4ae3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.693240 14619 tablet_service.cc:1467] Processing CreateTablet for tablet fe354ed675014a3192fc7fdc9937479c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 20:34:22.694595 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe354ed675014a3192fc7fdc9937479c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.697710 14618 tablet_service.cc:1467] Processing CreateTablet for tablet 3044f79ab3f74f788d8a263e141674a8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 20:34:22.698331 14460 tablet_service.cc:1467] Processing CreateTablet for tablet 5b527f3bc7bb43f88f1b17c445c3c0d0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 20:34:22.699051 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3044f79ab3f74f788d8a263e141674a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.699620 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b527f3bc7bb43f88f1b17c445c3c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.700950 14617 tablet_service.cc:1467] Processing CreateTablet for tablet f38960b1dd9245a8ba85fea601b09e94 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 20:34:22.702226 14617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38960b1dd9245a8ba85fea601b09e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.653548 14544 tablet_service.cc:1467] Processing CreateTablet for tablet fcae2b574a134bb781ba6c1a424f84a4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 20:34:22.703784 14467 tablet_service.cc:1467] Processing CreateTablet for tablet d8a0e4079ebe47aaa3c275ed7f559ab8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 20:34:22.704838 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38960b1dd9245a8ba85fea601b09e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.705050 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a0e4079ebe47aaa3c275ed7f559ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.705988 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.701830 14463 tablet_service.cc:1467] Processing CreateTablet for tablet 4248ec88bd4b44e990938a74fe3c4fdc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 20:34:22.707720 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4248ec88bd4b44e990938a74fe3c4fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.709813 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8986697653e048048cd32eb27e4ae3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.711045 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe354ed675014a3192fc7fdc9937479c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.712423 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4248ec88bd4b44e990938a74fe3c4fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.713737 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.715034 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3044f79ab3f74f788d8a263e141674a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.716351 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae2b574a134bb781ba6c1a424f84a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.718843 14616 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 20:34:22.721742 14616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.722237 14615 tablet_service.cc:1467] Processing CreateTablet for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 20:34:22.723816 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.725885 14614 tablet_service.cc:1467] Processing CreateTablet for tablet 5b527f3bc7bb43f88f1b17c445c3c0d0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 20:34:22.727217 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b527f3bc7bb43f88f1b17c445c3c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.746187 14459 tablet_service.cc:1467] Processing CreateTablet for tablet 289df1801e0045beb52c1aaaf43e73f0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 20:34:22.747709 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289df1801e0045beb52c1aaaf43e73f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.785224 14613 tablet_service.cc:1467] Processing CreateTablet for tablet 4248ec88bd4b44e990938a74fe3c4fdc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 20:34:22.787029 14613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4248ec88bd4b44e990938a74fe3c4fdc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:34:22.899039 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:22.868172 14459 tablet_service.cc:1467] Processing CreateTablet for tablet 7c10175264634fcd88e06ab41ee2a25c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 20:34:22.929116 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c10175264634fcd88e06ab41ee2a25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.934341 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 119ce96f4f7f49cfa853221635714edc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 20:34:22.935698 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119ce96f4f7f49cfa853221635714edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.937680 14465 tablet_service.cc:1467] Processing CreateTablet for tablet fb469b8694714b21bdf2c86de7aa2dad (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 20:34:22.938222 14461 tablet_service.cc:1467] Processing CreateTablet for tablet 17eec21790894c559ad9623765626416 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 20:34:22.938997 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb469b8694714b21bdf2c86de7aa2dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.952236 14674 tablet_bootstrap.cc:492] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:22.952764 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17eec21790894c559ad9623765626416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.964781 14462 tablet_service.cc:1467] Processing CreateTablet for tablet d2ced25ce9f1484081a8671550489f9b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 20:34:22.966267 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ced25ce9f1484081a8671550489f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.976965 14459 tablet_service.cc:1467] Processing CreateTablet for tablet b84df1548a8f46c492d5379377b5bc29 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 20:34:22.980664 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84df1548a8f46c492d5379377b5bc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.984678 14463 tablet_service.cc:1467] Processing CreateTablet for tablet f5277702c39f453cb1063cf506183484 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 20:34:22.986125 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5277702c39f453cb1063cf506183484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.987035 14537 tablet_service.cc:1467] Processing CreateTablet for tablet 7c10175264634fcd88e06ab41ee2a25c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 20:34:22.987464 14536 tablet_service.cc:1467] Processing CreateTablet for tablet d8a0e4079ebe47aaa3c275ed7f559ab8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 20:34:22.988175 14465 tablet_service.cc:1467] Processing CreateTablet for tablet cfcc7f43dcc54dc48f61c69d3d4f99a3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 20:34:22.997287 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcc7f43dcc54dc48f61c69d3d4f99a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:22.997854 14622 tablet_service.cc:1467] Processing CreateTablet for tablet d8a0e4079ebe47aaa3c275ed7f559ab8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 20:34:22.998119 14545 tablet_service.cc:1467] Processing CreateTablet for tablet 289df1801e0045beb52c1aaaf43e73f0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 20:34:22.998322 14540 tablet_service.cc:1467] Processing CreateTablet for tablet 119ce96f4f7f49cfa853221635714edc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 20:34:22.999617 14675 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:23.000193 14613 tablet_service.cc:1467] Processing CreateTablet for tablet 289df1801e0045beb52c1aaaf43e73f0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 20:34:23.001387 14621 tablet_service.cc:1467] Processing CreateTablet for tablet 7c10175264634fcd88e06ab41ee2a25c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 20:34:23.003916 14619 tablet_service.cc:1467] Processing CreateTablet for tablet 119ce96f4f7f49cfa853221635714edc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 20:34:23.006488 14541 tablet_service.cc:1467] Processing CreateTablet for tablet fb469b8694714b21bdf2c86de7aa2dad (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 20:34:23.007712 14468 tablet_service.cc:1467] Processing CreateTablet for tablet d8933c210fdc40cab4369e6aa56f7020 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 20:34:23.008956 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8933c210fdc40cab4369e6aa56f7020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.009438 14464 tablet_service.cc:1467] Processing CreateTablet for tablet dc270596f078400e9e1b181a9706b516 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 20:34:23.011364 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc270596f078400e9e1b181a9706b516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.014773 14674 tablet_bootstrap.cc:654] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:22.988382 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c10175264634fcd88e06ab41ee2a25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.036893 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119ce96f4f7f49cfa853221635714edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.040179 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a0e4079ebe47aaa3c275ed7f559ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.041023 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb469b8694714b21bdf2c86de7aa2dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.041738 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289df1801e0045beb52c1aaaf43e73f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.053901 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a0e4079ebe47aaa3c275ed7f559ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.054935 14620 tablet_service.cc:1467] Processing CreateTablet for tablet fb469b8694714b21bdf2c86de7aa2dad (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 20:34:23.055049 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 17eec21790894c559ad9623765626416 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 20:34:23.055049 14462 tablet_service.cc:1467] Processing CreateTablet for tablet 07ca0af1acba4a7ea01bf15f5f887457 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 20:34:23.056245 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb469b8694714b21bdf2c86de7aa2dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.057289 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ca0af1acba4a7ea01bf15f5f887457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.062479 14461 tablet_service.cc:1467] Processing CreateTablet for tablet 568833ac14f74d16a95723bbfa721a96 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 20:34:23.063995 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568833ac14f74d16a95723bbfa721a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.064141 14675 tablet_bootstrap.cc:654] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:23.067692 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119ce96f4f7f49cfa853221635714edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.068382 14613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289df1801e0045beb52c1aaaf43e73f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.070777 14676 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:23.070596 14614 tablet_service.cc:1467] Processing CreateTablet for tablet d8933c210fdc40cab4369e6aa56f7020 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 20:34:23.071377 14616 tablet_service.cc:1467] Processing CreateTablet for tablet f5277702c39f453cb1063cf506183484 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 20:34:23.073439 14617 tablet_service.cc:1467] Processing CreateTablet for tablet b84df1548a8f46c492d5379377b5bc29 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 20:34:23.073900 14618 tablet_service.cc:1467] Processing CreateTablet for tablet d2ced25ce9f1484081a8671550489f9b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 20:34:23.076227 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ced25ce9f1484081a8671550489f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.076912 14616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5277702c39f453cb1063cf506183484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.080749 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8933c210fdc40cab4369e6aa56f7020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.083890 14617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84df1548a8f46c492d5379377b5bc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.073854 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 58bf5720d4f5403b877975863b1cd46c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 20:34:23.088820 14538 tablet_service.cc:1467] Processing CreateTablet for tablet d2ced25ce9f1484081a8671550489f9b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 20:34:23.091029 14615 tablet_service.cc:1467] Processing CreateTablet for tablet 17eec21790894c559ad9623765626416 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 20:34:23.092317 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17eec21790894c559ad9623765626416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.056411 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17eec21790894c559ad9623765626416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.107532 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ced25ce9f1484081a8671550489f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.109453 14462 tablet_service.cc:1467] Processing CreateTablet for tablet fefa7fe42a5543c3970808c8a8e8e45c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 20:34:23.106966 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c10175264634fcd88e06ab41ee2a25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.117136 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefa7fe42a5543c3970808c8a8e8e45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.121272 14460 tablet_service.cc:1467] Processing CreateTablet for tablet 58bf5720d4f5403b877975863b1cd46c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 20:34:23.124869 14466 tablet_service.cc:1467] Processing CreateTablet for tablet ac95ea9d55914ec6912deffb9a91d8f8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 20:34:23.133854 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bf5720d4f5403b877975863b1cd46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.136742 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bf5720d4f5403b877975863b1cd46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.136855 14504 heartbeater.cc:503] Master 127.12.248.190:39439 was elected leader, sending a full tablet report...
I20260806 20:34:23.141090 14539 tablet_service.cc:1467] Processing CreateTablet for tablet d8933c210fdc40cab4369e6aa56f7020 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 20:34:23.142210 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac95ea9d55914ec6912deffb9a91d8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.138748 14674 tablet_bootstrap.cc:492] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:23.144438 14674 ts_tablet_manager.cc:1397] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.193s user 0.027s sys 0.022s
I20260806 20:34:23.150630 14674 raft_consensus.cc:348] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.151904 14674 raft_consensus.cc:374] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.152314 14674 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.153342 14674 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.151010 14459 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d79d20363456d80521feddfec49bc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 20:34:23.174901 14620 tablet_service.cc:1467] Processing CreateTablet for tablet cfcc7f43dcc54dc48f61c69d3d4f99a3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 20:34:23.180670 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8933c210fdc40cab4369e6aa56f7020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.181490 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcc7f43dcc54dc48f61c69d3d4f99a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.185117 14541 tablet_service.cc:1467] Processing CreateTablet for tablet b84df1548a8f46c492d5379377b5bc29 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 20:34:23.186594 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84df1548a8f46c492d5379377b5bc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.187400 14675 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:23.187899 14675 ts_tablet_manager.cc:1397] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.189s user 0.005s sys 0.007s
I20260806 20:34:23.193988 14622 tablet_service.cc:1467] Processing CreateTablet for tablet dc270596f078400e9e1b181a9706b516 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 20:34:23.188903 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d79d20363456d80521feddfec49bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.164397 14467 tablet_service.cc:1467] Processing CreateTablet for tablet a94bd288a57145e8b254611b1dd82586 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 20:34:23.210031 14545 tablet_service.cc:1467] Processing CreateTablet for tablet f5277702c39f453cb1063cf506183484 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 20:34:23.210032 14543 tablet_service.cc:1467] Processing CreateTablet for tablet cfcc7f43dcc54dc48f61c69d3d4f99a3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 20:34:23.211736 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5277702c39f453cb1063cf506183484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.212489 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcc7f43dcc54dc48f61c69d3d4f99a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.195399 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc270596f078400e9e1b181a9706b516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.219007 14463 tablet_service.cc:1467] Processing CreateTablet for tablet ef11d9fbd5a449aca4e91c72a6e73290 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 20:34:23.221210 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94bd288a57145e8b254611b1dd82586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.241806 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef11d9fbd5a449aca4e91c72a6e73290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.249485 14464 tablet_service.cc:1467] Processing CreateTablet for tablet 370a97aea0534403aeb11fe85b5a1ca9 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 20:34:23.250943 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370a97aea0534403aeb11fe85b5a1ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.252086 14675 raft_consensus.cc:348] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.252876 14675 raft_consensus.cc:374] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.253189 14675 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.261178 14674 ts_tablet_manager.cc:1428] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.116s user 0.010s sys 0.020s
I20260806 20:34:23.262466 14674 tablet_bootstrap.cc:492] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:23.264318 14618 tablet_service.cc:1467] Processing CreateTablet for tablet 07ca0af1acba4a7ea01bf15f5f887457 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 20:34:23.274906 14614 tablet_service.cc:1467] Processing CreateTablet for tablet fefa7fe42a5543c3970808c8a8e8e45c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 20:34:23.271004 14676 tablet_bootstrap.cc:654] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:23.280730 14359 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 121775us
I20260806 20:34:23.282418 14616 tablet_service.cc:1467] Processing CreateTablet for tablet 58bf5720d4f5403b877975863b1cd46c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
W20260806 20:34:23.283571 14356 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 72678us
I20260806 20:34:23.283390 14675 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.288074 14465 tablet_service.cc:1467] Processing CreateTablet for tablet 67e455c79bfc43e9ac867cf6719e289c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 20:34:23.298048 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e455c79bfc43e9ac867cf6719e289c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.290877 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefa7fe42a5543c3970808c8a8e8e45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.267993 14613 tablet_service.cc:1467] Processing CreateTablet for tablet 568833ac14f74d16a95723bbfa721a96 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 20:34:23.293829 14582 heartbeater.cc:503] Master 127.12.248.190:39439 was elected leader, sending a full tablet report...
I20260806 20:34:23.303366 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ca0af1acba4a7ea01bf15f5f887457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.308005 14613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568833ac14f74d16a95723bbfa721a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.319504 14540 tablet_service.cc:1467] Processing CreateTablet for tablet dc270596f078400e9e1b181a9706b516 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 20:34:23.322969 14615 tablet_service.cc:1467] Processing CreateTablet for tablet bef4a42f075d4ee1ac14ba3159109a18 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 20:34:23.323990 14616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bf5720d4f5403b877975863b1cd46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.326169 14658 heartbeater.cc:503] Master 127.12.248.190:39439 was elected leader, sending a full tablet report...
I20260806 20:34:23.327718 14674 tablet_bootstrap.cc:654] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:23.339764 14462 tablet_service.cc:1467] Processing CreateTablet for tablet 56e9ab616aca4e6bb97d457694148205 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 20:34:23.343010 14617 tablet_service.cc:1467] Processing CreateTablet for tablet ac95ea9d55914ec6912deffb9a91d8f8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 20:34:23.345113 14675 ts_tablet_manager.cc:1428] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.157s user 0.008s sys 0.011s
I20260806 20:34:23.346218 14675 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:23.347765 14619 tablet_service.cc:1467] Processing CreateTablet for tablet a94bd288a57145e8b254611b1dd82586 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 20:34:23.350564 14676 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:23.351068 14676 ts_tablet_manager.cc:1397] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.281s user 0.007s sys 0.018s
I20260806 20:34:23.352732 14460 tablet_service.cc:1467] Processing CreateTablet for tablet 53fd310e10544cb0af683409f6d8e454 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 20:34:23.355186 14676 raft_consensus.cc:348] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.355937 14676 raft_consensus.cc:374] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.356865 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef4a42f075d4ee1ac14ba3159109a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.360800 14617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac95ea9d55914ec6912deffb9a91d8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.363481 14621 tablet_service.cc:1467] Processing CreateTablet for tablet ef11d9fbd5a449aca4e91c72a6e73290 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
W20260806 20:34:23.363925 14505 tablet.cc:2369] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:34:23.364849 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef11d9fbd5a449aca4e91c72a6e73290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.367897 14536 tablet_service.cc:1467] Processing CreateTablet for tablet 07ca0af1acba4a7ea01bf15f5f887457 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 20:34:23.369643 14675 tablet_bootstrap.cc:654] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:23.370901 14620 tablet_service.cc:1467] Processing CreateTablet for tablet 370a97aea0534403aeb11fe85b5a1ca9 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 20:34:23.372198 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370a97aea0534403aeb11fe85b5a1ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.356233 14676 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.373201 14676 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.376838 14468 tablet_service.cc:1467] Processing CreateTablet for tablet bef4a42f075d4ee1ac14ba3159109a18 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 20:34:23.380599 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e9ab616aca4e6bb97d457694148205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.381304 14622 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d79d20363456d80521feddfec49bc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 20:34:23.382670 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fd310e10544cb0af683409f6d8e454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.371011 14461 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b00d6f3484ddbac8e7f8c2bdebba8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 20:34:23.387655 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b00d6f3484ddbac8e7f8c2bdebba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.387985 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc270596f078400e9e1b181a9706b516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.394060 14537 tablet_service.cc:1467] Processing CreateTablet for tablet 568833ac14f74d16a95723bbfa721a96 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 20:34:23.401286 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d79d20363456d80521feddfec49bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.404233 14674 tablet_bootstrap.cc:492] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:23.404768 14674 ts_tablet_manager.cc:1397] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.143s user 0.006s sys 0.007s
I20260806 20:34:23.404383 14543 tablet_service.cc:1467] Processing CreateTablet for tablet fefa7fe42a5543c3970808c8a8e8e45c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 20:34:23.405916 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefa7fe42a5543c3970808c8a8e8e45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.407581 14674 raft_consensus.cc:348] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.410284 14538 tablet_service.cc:1467] Processing CreateTablet for tablet ac95ea9d55914ec6912deffb9a91d8f8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 20:34:23.410575 14463 tablet_service.cc:1467] Processing CreateTablet for tablet a187485d6b5e4d9a9a9feafd48522726 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 20:34:23.411619 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac95ea9d55914ec6912deffb9a91d8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.411770 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187485d6b5e4d9a9a9feafd48522726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.416764 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94bd288a57145e8b254611b1dd82586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.418593 14539 tablet_service.cc:1467] Processing CreateTablet for tablet a94bd288a57145e8b254611b1dd82586 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 20:34:23.419384 14676 ts_tablet_manager.cc:1428] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.068s user 0.008s sys 0.000s
I20260806 20:34:23.419865 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94bd288a57145e8b254611b1dd82586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.420403 14676 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:23.421067 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568833ac14f74d16a95723bbfa721a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.423237 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ca0af1acba4a7ea01bf15f5f887457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.442654 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d79d20363456d80521feddfec49bc (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 20:34:23.443964 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d79d20363456d80521feddfec49bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.443638 14545 tablet_service.cc:1467] Processing CreateTablet for tablet ef11d9fbd5a449aca4e91c72a6e73290 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 20:34:23.447032 14467 tablet_service.cc:1467] Processing CreateTablet for tablet 87adab531acb4032af0c9a18dd9893e0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 20:34:23.450479 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 370a97aea0534403aeb11fe85b5a1ca9 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 20:34:23.428402 14676 tablet_bootstrap.cc:654] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:23.454313 14465 tablet_service.cc:1467] Processing CreateTablet for tablet 500956bd1a3846a3a7b16c51ddecbc44 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 20:34:23.455648 14679 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:23.456142 14679 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.458629 14679 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:23.408368 14674 raft_consensus.cc:374] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.461295 14675 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:23.481623 14681 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:23.482271 14681 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.483279 14620 tablet_service.cc:1467] Processing CreateTablet for tablet 56e9ab616aca4e6bb97d457694148205 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 20:34:23.471830 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef4a42f075d4ee1ac14ba3159109a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.465662 14674 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.499225 14613 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b00d6f3484ddbac8e7f8c2bdebba8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 20:34:23.502043 14614 tablet_service.cc:1467] Processing CreateTablet for tablet 67e455c79bfc43e9ac867cf6719e289c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 20:34:23.485006 14675 ts_tablet_manager.cc:1397] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.001s
I20260806 20:34:23.513998 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 67e455c79bfc43e9ac867cf6719e289c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 20:34:23.517341 14681 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:23.484623 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e9ab616aca4e6bb97d457694148205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.459916 14460 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8aa43086db48d2821816b1223056dd (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 20:34:23.526679 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87adab531acb4032af0c9a18dd9893e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.519524 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef11d9fbd5a449aca4e91c72a6e73290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.542770 14616 tablet_service.cc:1467] Processing CreateTablet for tablet 53fd310e10544cb0af683409f6d8e454 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 20:34:23.554574 14613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b00d6f3484ddbac8e7f8c2bdebba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.554327 14618 tablet_service.cc:1467] Processing CreateTablet for tablet a187485d6b5e4d9a9a9feafd48522726 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 20:34:23.556375 14537 tablet_service.cc:1467] Processing CreateTablet for tablet 53fd310e10544cb0af683409f6d8e454 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 20:34:23.562533 14459 tablet_service.cc:1467] Processing CreateTablet for tablet cf24df4ca64846378f216b30e63e27f0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 20:34:23.580211 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e455c79bfc43e9ac867cf6719e289c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.566798 14675 raft_consensus.cc:348] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.580833 14675 raft_consensus.cc:374] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.581234 14675 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.581750 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187485d6b5e4d9a9a9feafd48522726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.582086 14675 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.560253 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 3694b5a404a94fbea69b831722e2d3e4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 20:34:23.585528 14615 tablet_service.cc:1467] Processing CreateTablet for tablet 87adab531acb4032af0c9a18dd9893e0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 20:34:23.586756 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87adab531acb4032af0c9a18dd9893e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.570466 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500956bd1a3846a3a7b16c51ddecbc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.573441 14464 tablet_service.cc:1467] Processing CreateTablet for tablet db55591983ce42868e62df33ee1b56be (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 20:34:23.575318 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e455c79bfc43e9ac867cf6719e289c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.548646 14674 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.593207 14617 tablet_service.cc:1467] Processing CreateTablet for tablet 90e4ac7b6cd245aead7963fa7cbc5d2e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 20:34:23.593875 14674 ts_tablet_manager.cc:1428] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.189s user 0.006s sys 0.001s
I20260806 20:34:23.594499 14617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e4ac7b6cd245aead7963fa7cbc5d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.594863 14674 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:23.596753 14620 tablet_service.cc:1467] Processing CreateTablet for tablet 500956bd1a3846a3a7b16c51ddecbc44 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 20:34:23.597990 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500956bd1a3846a3a7b16c51ddecbc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.597853 14621 tablet_service.cc:1467] Processing CreateTablet for tablet 3694b5a404a94fbea69b831722e2d3e4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 20:34:23.605216 14676 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:23.605684 14676 ts_tablet_manager.cc:1397] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.185s user 0.011s sys 0.019s
I20260806 20:34:23.607966 14675 ts_tablet_manager.cc:1428] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.084s user 0.004s sys 0.003s
I20260806 20:34:23.608620 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370a97aea0534403aeb11fe85b5a1ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.610785 14675 tablet_bootstrap.cc:492] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:23.610852 14536 tablet_service.cc:1467] Processing CreateTablet for tablet 56e9ab616aca4e6bb97d457694148205 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 20:34:23.612365 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e9ab616aca4e6bb97d457694148205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.615859 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694b5a404a94fbea69b831722e2d3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.619468 14461 tablet_service.cc:1467] Processing CreateTablet for tablet e391f934d63c4ad788870d7abc39a79c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 20:34:23.621182 14462 tablet_service.cc:1467] Processing CreateTablet for tablet 90e4ac7b6cd245aead7963fa7cbc5d2e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 20:34:23.621922 14619 tablet_service.cc:1467] Processing CreateTablet for tablet db55591983ce42868e62df33ee1b56be (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 20:34:23.622859 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fd310e10544cb0af683409f6d8e454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.623248 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55591983ce42868e62df33ee1b56be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.624794 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694b5a404a94fbea69b831722e2d3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.629840 14540 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b00d6f3484ddbac8e7f8c2bdebba8 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 20:34:23.631315 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b00d6f3484ddbac8e7f8c2bdebba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.632822 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf24df4ca64846378f216b30e63e27f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.635363 14622 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8aa43086db48d2821816b1223056dd (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 20:34:23.636758 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8aa43086db48d2821816b1223056dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.636814 14679 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:23.608315 14676 raft_consensus.cc:348] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.645164 14676 raft_consensus.cc:374] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.645874 14676 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.644250 14679 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.647698 14679 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:23.648737 14676 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.651072 14542 tablet_service.cc:1467] Processing CreateTablet for tablet a187485d6b5e4d9a9a9feafd48522726 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 20:34:23.652112 14676 ts_tablet_manager.cc:1428] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.046s user 0.004s sys 0.002s
I20260806 20:34:23.653314 14676 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:23.655400 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187485d6b5e4d9a9a9feafd48522726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.661314 14675 tablet_bootstrap.cc:654] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:23.663288 14616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fd310e10544cb0af683409f6d8e454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.676640 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 87adab531acb4032af0c9a18dd9893e0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 20:34:23.678150 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87adab531acb4032af0c9a18dd9893e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.629884 14539 tablet_service.cc:1467] Processing CreateTablet for tablet bef4a42f075d4ee1ac14ba3159109a18 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 20:34:23.681505 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef4a42f075d4ee1ac14ba3159109a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.687510 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55591983ce42868e62df33ee1b56be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.689730 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8aa43086db48d2821816b1223056dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.690552 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e391f934d63c4ad788870d7abc39a79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.691205 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e4ac7b6cd245aead7963fa7cbc5d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.698397 14676 tablet_bootstrap.cc:654] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:23.701295 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:23.703524 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:23.705020 14442 leader_election.cc:341] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:23.706027 14442 leader_election.cc:341] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.711329 14620 tablet_service.cc:1467] Processing CreateTablet for tablet 6b370f85a8b944198972759bc1eb8061 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 20:34:23.711925 14674 tablet_bootstrap.cc:654] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:23.636760 14463 tablet_service.cc:1467] Processing CreateTablet for tablet 6b370f85a8b944198972759bc1eb8061 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 20:34:23.713486 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b370f85a8b944198972759bc1eb8061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.713894 14538 tablet_service.cc:1467] Processing CreateTablet for tablet 500956bd1a3846a3a7b16c51ddecbc44 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 20:34:23.715981 14545 tablet_service.cc:1467] Processing CreateTablet for tablet 3694b5a404a94fbea69b831722e2d3e4 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 20:34:23.717170 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694b5a404a94fbea69b831722e2d3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.719453 14683 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:23.719908 14683 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.733892 14683 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:23.747980 14543 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 24) took 1009 ms. Trace:
I20260806 20:34:23.749814 14465 tablet_service.cc:1467] Processing CreateTablet for tablet 7815d5557ccf4e569039960d0a8f24ba (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 20:34:23.749452 14543 rpcz_store.cc:276] 0806 20:34:22.738716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.403503 (+664787us) service_pool.cc:227] Handling call
0806 20:34:23.405415 (+ 1912us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.405567 (+ 152us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.614213 (+208646us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.721704 (+107491us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.747920 (+ 26216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:23.751257 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7815d5557ccf4e569039960d0a8f24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.752810 14543 tablet_service.cc:1467] Processing CreateTablet for tablet db55591983ce42868e62df33ee1b56be (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 20:34:23.749117 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500956bd1a3846a3a7b16c51ddecbc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.764583 14683 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:23.784479 14616 tablet_service.cc:1467] Processing CreateTablet for tablet 7815d5557ccf4e569039960d0a8f24ba (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 20:34:23.785077 14618 tablet_service.cc:1467] Processing CreateTablet for tablet e391f934d63c4ad788870d7abc39a79c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 20:34:23.786494 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:23.788619 14595 leader_election.cc:341] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.765249 14683 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.802414 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55591983ce42868e62df33ee1b56be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.804243 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 90e4ac7b6cd245aead7963fa7cbc5d2e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 20:34:23.805564 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e4ac7b6cd245aead7963fa7cbc5d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.810141 14468 tablet_service.cc:1467] Processing CreateTablet for tablet d01e9209cd924024ae8e7e630b5accd3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 20:34:23.811415 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01e9209cd924024ae8e7e630b5accd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.813669 14537 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 32) took 1044 ms. Trace:
I20260806 20:34:23.814039 14537 rpcz_store.cc:276] 0806 20:34:22.769120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.539613 (+770493us) service_pool.cc:227] Handling call
0806 20:34:23.622420 (+ 82807us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.622510 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.632620 (+ 10110us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.813178 (+180558us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.813609 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:23.814968 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:23.815857 14537 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8aa43086db48d2821816b1223056dd (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 20:34:23.816118 14614 tablet_service.cc:1467] Processing CreateTablet for tablet cf24df4ca64846378f216b30e63e27f0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 20:34:23.816921 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:23.817971 14444 leader_election.cc:341] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.843030 14444 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:23.827594 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 05f645749c8846cf8c2f4dd196506e4d (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 20:34:23.827883 14462 tablet_service.cc:1467] Processing CreateTablet for tablet 1e40012e131a4bfe9d61a3653f3298c6 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 20:34:23.826308 14461 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7667fe05a410bbd313d3d601d18b2 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 20:34:23.828924 14541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 31) took 1064 ms. Trace:
I20260806 20:34:23.829993 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:23.852746 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e40012e131a4bfe9d61a3653f3298c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.854172 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7667fe05a410bbd313d3d601d18b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.831961 14683 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:23.859194 14460 tablet_service.cc:1467] Processing CreateTablet for tablet 08d2a67d79c54f588be9e64dce5b2202 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 20:34:23.860033 14539 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 35) took 1078 ms. Trace:
I20260806 20:34:23.843858 14679 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:23.847574 14542 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 36) took 1023 ms. Trace:
I20260806 20:34:23.860424 14539 rpcz_store.cc:276] 0806 20:34:22.782152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.629042 (+846890us) service_pool.cc:227] Handling call
0806 20:34:23.680991 (+ 51949us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.681087 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.785054 (+103967us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.857099 (+ 72045us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.859979 (+ 2880us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:34:23.845026 14444 leader_election.cc:341] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.862166 14444 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:23.862260 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 6b370f85a8b944198972759bc1eb8061 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 20:34:23.846433 14681 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:23.864244 14681 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.864995 14679 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:23.860847 14542 rpcz_store.cc:276] 0806 20:34:22.824276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.649101 (+824825us) service_pool.cc:227] Handling call
0806 20:34:23.651986 (+ 2885us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.652157 (+ 171us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.712406 (+ 60249us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.844781 (+132375us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.847148 (+ 2367us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593e82 00007fb9b94ac8b9 00007fb9b83bf5d8 00007fb9b8483063 00007fb9b847ee2f 00007fb9b847f5ed 00007fb9b8430869 00007fb9b1a32c44 00007fb9b1a32bd2
0806 20:34:23.847516 (+ 368us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3543808}
I20260806 20:34:23.866199 14681 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:23.866540 14622 tablet_service.cc:1467] Processing CreateTablet for tablet 05f645749c8846cf8c2f4dd196506e4d (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 20:34:23.867405 14542 tablet_service.cc:1467] Processing CreateTablet for tablet e391f934d63c4ad788870d7abc39a79c (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 20:34:23.867865 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:23.869243 14595 leader_election.cc:341] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.845718 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8aa43086db48d2821816b1223056dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.877163 14464 tablet_service.cc:1467] Processing CreateTablet for tablet c7a9c839b021436dac626c7630f18f8e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
W20260806 20:34:23.857862 14596 leader_election.cc:341] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.878013 14596 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:23.858134 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f645749c8846cf8c2f4dd196506e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.878950 14681 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:23.882436 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:23.835362 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b370f85a8b944198972759bc1eb8061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.883538 14463 tablet_service.cc:1467] Processing CreateTablet for tablet 53bd3d0855da421aac4540ae1bfbb8f3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 20:34:23.852238 14541 rpcz_store.cc:276] 0806 20:34:22.764808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.513111 (+748303us) service_pool.cc:227] Handling call
0806 20:34:23.539835 (+ 26724us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.561813 (+ 21978us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593fee 00007fb9b94ac8f9 00007fb9b83bf6a1 00007fb9b847f3e9 00007fb9b8430869 00007fb9b1a32c44 00007fb9b1a32bd2 00007fb9b1a32b01 00007fb9b1a32a7c
0806 20:34:23.561894 (+ 81us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.725341 (+163447us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.797336 (+ 71995us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.828868 (+ 31532us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4226432}
I20260806 20:34:23.884765 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bd3d0855da421aac4540ae1bfbb8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.831677 14615 tablet_service.cc:1467] Processing CreateTablet for tablet d01e9209cd924024ae8e7e630b5accd3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 20:34:23.885602 14616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7815d5557ccf4e569039960d0a8f24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.886231 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 7815d5557ccf4e569039960d0a8f24ba (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20260806 20:34:23.887496 14596 leader_election.cc:341] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.887926 14596 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:23.880872 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b370f85a8b944198972759bc1eb8061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.888309 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e391f934d63c4ad788870d7abc39a79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.890264 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf24df4ca64846378f216b30e63e27f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.873624 14613 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7667fe05a410bbd313d3d601d18b2 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 20:34:23.891592 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e391f934d63c4ad788870d7abc39a79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.893115 14681 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:23.893383 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d2a67d79c54f588be9e64dce5b2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.859192 14465 tablet_service.cc:1467] Processing CreateTablet for tablet 7f41753baced49a1997940dfc28b2e00 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 20:34:23.893889 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7815d5557ccf4e569039960d0a8f24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.894352 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f41753baced49a1997940dfc28b2e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.877235 14459 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7dc7fd91be46ecb624803a9b029e5b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 20:34:23.895726 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7dc7fd91be46ecb624803a9b029e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.907385 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f645749c8846cf8c2f4dd196506e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.908960 14621 tablet_service.cc:1467] Processing CreateTablet for tablet 1e40012e131a4bfe9d61a3653f3298c6 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 20:34:23.912845 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a9c839b021436dac626c7630f18f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.862936 14675 tablet_bootstrap.cc:492] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:23.911065 14613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7667fe05a410bbd313d3d601d18b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.860236 14536 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 33) took 1087 ms. Trace:
W20260806 20:34:23.921227 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:23.927903 14543 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 40) took 1079 ms. Trace:
I20260806 20:34:23.934675 14540 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 34) took 1156 ms. Trace:
I20260806 20:34:23.918303 14536 rpcz_store.cc:276] 0806 20:34:22.773247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.610013 (+836766us) service_pool.cc:227] Handling call
0806 20:34:23.611898 (+ 1885us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.612001 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.791819 (+179818us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.859784 (+ 67965us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.860187 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:23.937183 14538 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 38) took 1105 ms. Trace:
I20260806 20:34:23.939247 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:23.940215 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:23.940896 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e40012e131a4bfe9d61a3653f3298c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:34:23.977911 14518 leader_election.cc:341] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.943204 14674 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:23.978258 14468 tablet_service.cc:1467] Processing CreateTablet for tablet eb32d3aa416443529139823cf8e2d00b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 20:34:23.979059 14674 ts_tablet_manager.cc:1397] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.384s user 0.009s sys 0.016s
I20260806 20:34:23.979481 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb32d3aa416443529139823cf8e2d00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.981181 14537 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 42) took 1055 ms. Trace:
I20260806 20:34:23.981905 14674 raft_consensus.cc:348] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.982674 14674 raft_consensus.cc:374] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.982971 14674 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.983645 14674 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.919659 14675 ts_tablet_manager.cc:1397] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.309s user 0.010s sys 0.026s
I20260806 20:34:23.987084 14675 raft_consensus.cc:348] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.929314 14619 tablet_service.cc:1467] Processing CreateTablet for tablet 7f41753baced49a1997940dfc28b2e00 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 20:34:23.988729 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f41753baced49a1997940dfc28b2e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.990314 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 457a5ba876d14681a3fa2338fd9f6c4e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 20:34:23.991545 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457a5ba876d14681a3fa2338fd9f6c4e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:34:23.992208 14518 leader_election.cc:341] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:23.987795 14675 raft_consensus.cc:374] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:23.993281 14675 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:23.981541 14537 rpcz_store.cc:276] 0806 20:34:22.925733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.814980 (+889247us) service_pool.cc:227] Handling call
0806 20:34:23.845244 (+ 30264us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.845345 (+ 101us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.881863 (+ 36518us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.952337 (+ 70474us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.981125 (+ 28788us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:23.997025 14675 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:23.949112 14536 tablet_service.cc:1467] Processing CreateTablet for tablet cf24df4ca64846378f216b30e63e27f0 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 20:34:23.945091 14543 rpcz_store.cc:276] 0806 20:34:22.848746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.751767 (+903021us) service_pool.cc:227] Handling call
0806 20:34:23.753760 (+ 1993us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.753857 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.841898 (+ 88041us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.927383 (+ 85485us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.927844 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:24.001401 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7667fe05a410bbd313d3d601d18b2 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 20:34:24.002993 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7667fe05a410bbd313d3d601d18b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.966856 14542 tablet_service.cc:1467] Processing CreateTablet for tablet d01e9209cd924024ae8e7e630b5accd3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 20:34:24.005183 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01e9209cd924024ae8e7e630b5accd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.968537 14616 tablet_service.cc:1467] Processing CreateTablet for tablet 53bd3d0855da421aac4540ae1bfbb8f3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 20:34:24.006359 14537 tablet_service.cc:1467] Processing CreateTablet for tablet 05f645749c8846cf8c2f4dd196506e4d (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 20:34:24.006927 14616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bd3d0855da421aac4540ae1bfbb8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.007586 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f645749c8846cf8c2f4dd196506e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:23.914554 14617 tablet_service.cc:1467] Processing CreateTablet for tablet 08d2a67d79c54f588be9e64dce5b2202 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 20:34:23.975618 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:23.976392 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:24.018399 14521 leader_election.cc:341] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.019042 14521 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:23.974020 14538 rpcz_store.cc:276] 0806 20:34:22.832295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.713042 (+880747us) service_pool.cc:227] Handling call
0806 20:34:23.747518 (+ 34476us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.748500 (+ 982us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.848550 (+100050us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.936662 (+ 88112us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.937127 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10112}
I20260806 20:34:24.020756 14683 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.021566 14538 tablet_service.cc:1467] Processing CreateTablet for tablet 1e40012e131a4bfe9d61a3653f3298c6 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 20:34:24.016557 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf24df4ca64846378f216b30e63e27f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.018579 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01e9209cd924024ae8e7e630b5accd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.033131 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 08d2a67d79c54f588be9e64dce5b2202 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 20:34:23.976684 14676 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.034475 14617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d2a67d79c54f588be9e64dce5b2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.034708 14676 ts_tablet_manager.cc:1397] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.382s user 0.010s sys 0.010s
I20260806 20:34:23.946326 14540 rpcz_store.cc:276] 0806 20:34:22.777552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.628848 (+851296us) service_pool.cc:227] Handling call
0806 20:34:23.630797 (+ 1949us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.630905 (+ 108us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.798950 (+168045us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:23.933495 (+134545us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:23.933940 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:24.032932 14674 ts_tablet_manager.cc:1428] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20260806 20:34:24.038722 14540 tablet_service.cc:1467] Processing CreateTablet for tablet 7f41753baced49a1997940dfc28b2e00 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 20:34:24.042208 14674 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:23.970505 14467 tablet_service.cc:1467] Processing CreateTablet for tablet b16d0272c85445ed95a93660f8851656 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20260806 20:34:24.046442 14356 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 49144.2us
I20260806 20:34:24.047489 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16d0272c85445ed95a93660f8851656. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:34:24.052920 14521 leader_election.cc:341] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.053381 14521 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.055151 14676 raft_consensus.cc:348] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.055938 14676 raft_consensus.cc:374] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.056229 14676 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.056938 14676 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.059103 14676 ts_tablet_manager.cc:1428] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20260806 20:34:24.059576 14683 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.060587 14675 ts_tablet_manager.cc:1428] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.076s user 0.003s sys 0.005s
I20260806 20:34:24.061599 14675 tablet_bootstrap.cc:492] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.066285 14618 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38498 (request call id 44) took 1015 ms. Trace:
I20260806 20:34:24.066672 14618 rpcz_store.cc:276] 0806 20:34:23.050985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.741876 (+690891us) service_pool.cc:227] Handling call
0806 20:34:23.785900 (+ 44024us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.786024 (+ 124us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.970304 (+184280us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:24.065800 (+ 95496us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:24.066228 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:24.070932 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 53bd3d0855da421aac4540ae1bfbb8f3 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 20:34:24.072324 14620 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7dc7fd91be46ecb624803a9b029e5b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 20:34:24.073634 14620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7dc7fd91be46ecb624803a9b029e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.076153 14618 tablet_service.cc:1467] Processing CreateTablet for tablet c7a9c839b021436dac626c7630f18f8e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 20:34:24.077417 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a9c839b021436dac626c7630f18f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.078270 14613 tablet_service.cc:1467] Processing CreateTablet for tablet b16d0272c85445ed95a93660f8851656 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 20:34:24.081019 14622 tablet_service.cc:1467] Processing CreateTablet for tablet eb32d3aa416443529139823cf8e2d00b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 20:34:24.034475 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d2a67d79c54f588be9e64dce5b2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.092046 14676 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.093537 14536 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7dc7fd91be46ecb624803a9b029e5b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 20:34:24.116505 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bd3d0855da421aac4540ae1bfbb8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.118389 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f41753baced49a1997940dfc28b2e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.095343 14675 tablet_bootstrap.cc:654] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.100826 14545 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 39) took 1264 ms. Trace:
I20260806 20:34:24.120810 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e40012e131a4bfe9d61a3653f3298c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.120746 14545 rpcz_store.cc:276] 0806 20:34:22.836550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.715153 (+878603us) service_pool.cc:227] Handling call
0806 20:34:23.716734 (+ 1581us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.716828 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:23.994231 (+277403us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:24.082621 (+ 88390us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:24.100767 (+ 18146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:24.102764 14674 tablet_bootstrap.cc:654] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.122752 14545 tablet_service.cc:1467] Processing CreateTablet for tablet c7a9c839b021436dac626c7630f18f8e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 20:34:24.123996 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a9c839b021436dac626c7630f18f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.124971 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7dc7fd91be46ecb624803a9b029e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.128784 14613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16d0272c85445ed95a93660f8851656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.132634 14676 tablet_bootstrap.cc:654] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.134565 14621 tablet_service.cc:1467] Processing CreateTablet for tablet 457a5ba876d14681a3fa2338fd9f6c4e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 20:34:24.135818 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457a5ba876d14681a3fa2338fd9f6c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.141706 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb32d3aa416443529139823cf8e2d00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.142009 14544 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54486 (request call id 41) took 1287 ms. Trace:
I20260806 20:34:24.142423 14544 rpcz_store.cc:276] 0806 20:34:22.854847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:23.803325 (+948478us) service_pool.cc:227] Handling call
0806 20:34:23.805131 (+ 1806us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:34:23.805223 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:34:24.036097 (+230874us) tablet_metadata.cc:641] Metadata flushed
0806 20:34:24.141466 (+105369us) tablet_replica.cc:205] Creating consensus instance
0806 20:34:24.141950 (+ 484us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:34:24.145056 14544 tablet_service.cc:1467] Processing CreateTablet for tablet b16d0272c85445ed95a93660f8851656 (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 20:34:24.173267 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16d0272c85445ed95a93660f8851656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.158584 14543 tablet_service.cc:1467] Processing CreateTablet for tablet eb32d3aa416443529139823cf8e2d00b (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 20:34:24.162863 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 457a5ba876d14681a3fa2338fd9f6c4e (DEFAULT_TABLE table=test-workload [id=8a9bb69966b34eb8b32423b196259815]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 20:34:24.140424 14675 tablet_bootstrap.cc:492] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.174660 14675 ts_tablet_manager.cc:1397] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.000s
I20260806 20:34:24.177544 14675 raft_consensus.cc:348] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.178323 14675 raft_consensus.cc:374] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.178663 14675 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.179495 14675 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.182797 14675 ts_tablet_manager.cc:1428] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 20:34:24.189213 14675 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.185191 14674 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.189955 14674 ts_tablet_manager.cc:1397] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.148s user 0.015s sys 0.001s
I20260806 20:34:24.180915 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457a5ba876d14681a3fa2338fd9f6c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.193758 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb32d3aa416443529139823cf8e2d00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:24.198949 14676 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.199445 14676 ts_tablet_manager.cc:1397] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.019s
I20260806 20:34:24.201573 14674 raft_consensus.cc:348] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.202391 14674 raft_consensus.cc:374] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.202175 14676 raft_consensus.cc:348] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.207424 14674 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.207515 14676 raft_consensus.cc:374] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.207876 14676 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.208632 14676 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.212745 14674 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.213601 14676 ts_tablet_manager.cc:1428] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20260806 20:34:24.214560 14676 tablet_bootstrap.cc:492] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.218815 14681 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.219264 14681 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.220237 14676 tablet_bootstrap.cc:654] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.220443 14674 ts_tablet_manager.cc:1428] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.030s user 0.004s sys 0.002s
I20260806 20:34:24.221200 14681 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:24.221403 14674 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.223009 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.223640 14478 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:24.224858 14595 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:24.226828 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:24.232642 14596 leader_election.cc:341] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.238318 14674 tablet_bootstrap.cc:654] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.241962 14675 tablet_bootstrap.cc:654] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.243446 14681 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.243835 14681 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.244093 14674 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.244158 14681 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.244696 14674 ts_tablet_manager.cc:1397] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.003s
I20260806 20:34:24.245170 14683 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.245652 14683 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.247509 14683 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.248243 14674 raft_consensus.cc:348] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.249173 14675 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.249234 14674 raft_consensus.cc:374] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.249732 14675 ts_tablet_manager.cc:1397] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.000s
I20260806 20:34:24.249733 14674 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.249971 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.250680 14632 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:24.250838 14674 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.252017 14521 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:24.252681 14675 raft_consensus.cc:348] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.253132 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.253438 14675 raft_consensus.cc:374] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.253715 14675 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.254154 14674 ts_tablet_manager.cc:1428] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260806 20:34:24.254503 14518 leader_election.cc:341] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.254455 14675 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.255378 14674 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.258658 14679 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.259214 14679 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.263144 14679 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.263680 14674 tablet_bootstrap.cc:654] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.264992 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.266575 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.268608 14683 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.269074 14683 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.269476 14683 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:24.272843 14442 leader_election.cc:341] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:24.274187 14444 leader_election.cc:341] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.274721 14444 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.276520 14679 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.289497 14683 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.293907 14675 ts_tablet_manager.cc:1428] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.044s user 0.033s sys 0.007s
I20260806 20:34:24.294899 14675 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.296376 14681 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.296749 14683 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.298159 14681 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:24.300351 14674 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.292366 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:24.300853 14674 ts_tablet_manager.cc:1397] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.008s
I20260806 20:34:24.300828 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:24.301620 14675 tablet_bootstrap.cc:654] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:24.302260 14596 leader_election.cc:341] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.303486 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.304054 14632 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.303884 14674 raft_consensus.cc:348] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.304425 14676 tablet_bootstrap.cc:492] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.304666 14674 raft_consensus.cc:374] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.300397 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:24.305164 14676 ts_tablet_manager.cc:1397] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.091s user 0.036s sys 0.036s
I20260806 20:34:24.305128 14674 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.305575 14479 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.305914 14674 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.308182 14674 ts_tablet_manager.cc:1428] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 20:34:24.307941 14676 raft_consensus.cc:348] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.308674 14676 raft_consensus.cc:374] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.308940 14676 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.309085 14674 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.309751 14676 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.312263 14676 ts_tablet_manager.cc:1428] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260806 20:34:24.313247 14676 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.315321 14674 tablet_bootstrap.cc:654] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:24.317898 14518 leader_election.cc:341] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.321133 14676 tablet_bootstrap.cc:654] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.327392 14674 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.328006 14674 ts_tablet_manager.cc:1397] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.001s
I20260806 20:34:24.331593 14674 raft_consensus.cc:348] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.332620 14674 raft_consensus.cc:374] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.332978 14674 raft_consensus.cc:729] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.333950 14674 consensus_queue.cc:260] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.336874 14676 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.337375 14676 ts_tablet_manager.cc:1397] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.007s
I20260806 20:34:24.337353 14674 ts_tablet_manager.cc:1428] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.009s user 0.009s sys 0.002s
I20260806 20:34:24.338900 14674 tablet_bootstrap.cc:492] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.340703 14679 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.340435 14676 raft_consensus.cc:348] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.341274 14676 raft_consensus.cc:374] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.341575 14676 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.341147 14679 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.342286 14676 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.343732 14679 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.344802 14676 ts_tablet_manager.cc:1428] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:34:24.345827 14676 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.346181 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.347568 14674 tablet_bootstrap.cc:654] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:24.347826 14444 leader_election.cc:341] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.348863 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.349545 14555 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:24.350823 14442 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.351859 14676 tablet_bootstrap.cc:654] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.352087 14679 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.352416 14679 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.352788 14679 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.398295 14479 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:24.403863 14595 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:24.405097 14681 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.405773 14681 raft_consensus.cc:686] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:24.406900 14681 consensus_queue.cc:237] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.407047 14679 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.400254 14632 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:24.411070 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:24.411901 14555 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.413193 14521 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:24.414746 14683 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.433254 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.433480 14679 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:24.435372 14444 leader_election.cc:341] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.436378 14555 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:24.437876 14442 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.440876 14679 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.445428 14676 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.445794 14675 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.445955 14676 ts_tablet_manager.cc:1397] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.100s user 0.052s sys 0.045s
I20260806 20:34:24.446327 14683 raft_consensus.cc:686] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:24.447481 14674 tablet_bootstrap.cc:492] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.448184 14683 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.448850 14676 raft_consensus.cc:348] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.449620 14676 raft_consensus.cc:374] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.449949 14676 raft_consensus.cc:729] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.450373 14679 raft_consensus.cc:686] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:24.450819 14676 consensus_queue.cc:260] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.451547 14675 ts_tablet_manager.cc:1397] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.157s user 0.040s sys 0.102s
I20260806 20:34:24.451296 14679 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.453430 14676 ts_tablet_manager.cc:1428] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:34:24.448083 14674 ts_tablet_manager.cc:1397] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.109s user 0.019s sys 0.053s
I20260806 20:34:24.454360 14676 tablet_bootstrap.cc:492] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.454224 14675 raft_consensus.cc:348] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.454972 14675 raft_consensus.cc:374] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.455263 14675 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.455910 14675 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.456655 14674 raft_consensus.cc:348] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.456290 14372 catalog_manager.cc:5495] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:24.458349 14675 ts_tablet_manager.cc:1428] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:34:24.457316 14674 raft_consensus.cc:374] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.458719 14674 raft_consensus.cc:729] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.459311 14675 tablet_bootstrap.cc:492] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.459627 14674 consensus_queue.cc:260] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.461078 14676 tablet_bootstrap.cc:654] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.462122 14674 ts_tablet_manager.cc:1428] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 20:34:24.463277 14674 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.466815 14681 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.467360 14681 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.470357 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.470999 14479 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:24.472214 14595 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:24.473235 14707 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.473605 14707 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.473932 14707 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.474280 14370 catalog_manager.cc:5495] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:24.478747 14679 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.479230 14679 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.481447 14675 tablet_bootstrap.cc:654] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.482170 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.482867 14555 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:24.483770 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.482156 14707 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.484365 14632 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:24.485826 14442 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:24.486414 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.486865 14679 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.487619 14679 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.487942 14679 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:34:24.487911 14596 leader_election.cc:341] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.488610 14679 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.491047 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:34:24.492364 14596 leader_election.cc:341] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.488969 14681 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:24.493654 14707 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:24.497408 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:24.498035 14479 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.501217 14675 tablet_bootstrap.cc:492] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.501217 14676 tablet_bootstrap.cc:492] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.501827 14675 ts_tablet_manager.cc:1397] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.004s
I20260806 20:34:24.501912 14676 ts_tablet_manager.cc:1397] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.048s user 0.020s sys 0.024s
I20260806 20:34:24.504554 14676 raft_consensus.cc:348] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.504871 14675 raft_consensus.cc:348] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.505347 14676 raft_consensus.cc:374] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.505604 14675 raft_consensus.cc:374] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.505744 14676 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.505923 14675 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.505795 14371 catalog_manager.cc:5495] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:24.506793 14675 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.506727 14676 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.508968 14676 ts_tablet_manager.cc:1428] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260806 20:34:24.509303 14675 ts_tablet_manager.cc:1428] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 20:34:24.509963 14676 tablet_bootstrap.cc:492] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.510253 14675 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.510689 14716 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.511158 14716 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.514282 14479 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:24.514170 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.515450 14595 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:34:24.515596 14442 leader_election.cc:341] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.516100 14676 tablet_bootstrap.cc:654] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.516255 14675 tablet_bootstrap.cc:654] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.516258 14707 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.516991 14707 raft_consensus.cc:686] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:24.517913 14707 consensus_queue.cc:237] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.521286 14683 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.522617 14683 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.526678 14674 tablet_bootstrap.cc:654] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.496048 14679 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.529721 14683 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.530184 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.530475 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:24.533164 14518 leader_election.cc:341] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.535316 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.536180 14632 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
W20260806 20:34:24.532131 14444 leader_election.cc:341] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.537689 14444 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.539099 14521 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:24.542418 14683 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.542630 14679 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.542761 14683 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.543359 14683 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.543421 14679 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.544049 14679 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.545737 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:24.546298 14555 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.546607 14679 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.547315 14679 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.528735 14716 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.557531 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.560573 14676 tablet_bootstrap.cc:492] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.561018 14676 ts_tablet_manager.cc:1397] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.007s
I20260806 20:34:24.562057 14555 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
W20260806 20:34:24.560397 14442 leader_election.cc:341] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.563297 14442 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:24.563561 14676 raft_consensus.cc:348] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.564244 14676 raft_consensus.cc:374] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.564742 14676 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.565371 14676 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.566732 14683 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.567411 14676 ts_tablet_manager.cc:1428] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260806 20:34:24.568279 14676 tablet_bootstrap.cc:492] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.569504 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:34:24.570848 14518 leader_election.cc:341] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.574767 14683 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.574271 14679 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.576284 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.576577 14679 raft_consensus.cc:686] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:24.576913 14632 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.577448 14679 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:24.579506 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47404.6us
I20260806 20:34:24.581609 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.583303 14633 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.584367 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.581636 14675 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.586031 14675 ts_tablet_manager.cc:1397] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.031s
I20260806 20:34:24.588376 14632 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:24.588708 14675 raft_consensus.cc:348] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.589572 14675 raft_consensus.cc:374] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.589948 14675 raft_consensus.cc:729] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.590250 14521 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:24.590821 14675 consensus_queue.cc:260] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.591557 14683 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.592054 14683 raft_consensus.cc:686] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:24.592878 14683 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.594193 14675 ts_tablet_manager.cc:1428] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260806 20:34:24.595347 14675 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
W20260806 20:34:24.589164 14444 leader_election.cc:341] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.601821 14676 tablet_bootstrap.cc:654] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.602293 14444 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.606031 14728 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.606549 14633 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:24.612619 14675 tablet_bootstrap.cc:654] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.615028 14370 catalog_manager.cc:5495] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:24.616250 14728 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.616732 14728 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.618772 14683 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.619225 14683 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.619946 14674 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.620040 14728 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.620476 14674 ts_tablet_manager.cc:1397] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.017s
I20260806 20:34:24.621188 14683 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.622488 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.623077 14555 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:24.624388 14442 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:24.624979 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.625633 14479 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:24.626271 14679 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.626686 14679 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.626706 14676 tablet_bootstrap.cc:492] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.626920 14518 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:24.627185 14679 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.627339 14676 ts_tablet_manager.cc:1397] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.004s
I20260806 20:34:24.625561 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.627918 14633 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:24.629474 14683 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.629818 14683 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.630151 14683 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.630008 14676 raft_consensus.cc:348] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.630713 14676 raft_consensus.cc:374] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.630985 14676 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.631616 14676 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.635426 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.636113 14632 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:24.636425 14676 ts_tablet_manager.cc:1428] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260806 20:34:24.634656 14679 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.642504 14676 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.644374 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:24.644897 14675 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.645371 14675 ts_tablet_manager.cc:1397] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.050s user 0.016s sys 0.002s
I20260806 20:34:24.645433 14679 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.646687 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.647418 14632 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.647598 14674 raft_consensus.cc:348] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.647871 14675 raft_consensus.cc:348] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.648613 14674 raft_consensus.cc:374] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.648777 14675 raft_consensus.cc:374] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.649060 14674 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.649178 14675 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.649793 14674 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.639858 14683 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.650027 14675 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.651237 14555 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:24.653235 14683 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.654007 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:24.654769 14479 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:24.655215 14674 ts_tablet_manager.cc:1428] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.034s user 0.002s sys 0.004s
I20260806 20:34:24.656103 14674 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.657042 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.657964 14675 ts_tablet_manager.cc:1428] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 20:34:24.658823 14675 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.658595 14521 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.660077 14683 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.661816 14676 tablet_bootstrap.cc:654] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.666198 14675 tablet_bootstrap.cc:654] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.666724 14370 catalog_manager.cc:5495] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:24.663679 14674 tablet_bootstrap.cc:654] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.664626 14632 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:24.675232 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:24.676251 14679 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.676762 14679 raft_consensus.cc:686] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:24.677577 14679 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.681373 14675 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.681949 14675 ts_tablet_manager.cc:1397] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.001s
I20260806 20:34:24.685508 14675 raft_consensus.cc:348] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.686409 14675 raft_consensus.cc:374] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.686741 14675 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.686787 14372 catalog_manager.cc:5495] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:24.687685 14675 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.690851 14675 ts_tablet_manager.cc:1428] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 20:34:24.692332 14675 tablet_bootstrap.cc:492] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.703763 14674 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.703958 14675 tablet_bootstrap.cc:654] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.704789 14674 ts_tablet_manager.cc:1397] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.007s
I20260806 20:34:24.707302 14676 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.707638 14674 raft_consensus.cc:348] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.707767 14676 ts_tablet_manager.cc:1397] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.023s
I20260806 20:34:24.708437 14674 raft_consensus.cc:374] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.708750 14674 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.709474 14674 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.710719 14676 raft_consensus.cc:348] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.711366 14676 raft_consensus.cc:374] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.711623 14676 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.711727 14674 ts_tablet_manager.cc:1428] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 20:34:24.712251 14676 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.713356 14683 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.713785 14683 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.714367 14676 ts_tablet_manager.cc:1428] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:34:24.715098 14676 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.715796 14683 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.716871 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.716893 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.717657 14632 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:24.714381 14675 tablet_bootstrap.cc:492] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.718918 14675 ts_tablet_manager.cc:1397] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.004s
W20260806 20:34:24.718402 14518 leader_election.cc:341] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.718885 14674 tablet_bootstrap.cc:492] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.720702 14676 tablet_bootstrap.cc:654] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.721750 14675 raft_consensus.cc:348] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.722494 14675 raft_consensus.cc:374] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.722749 14675 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.722771 14521 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:24.724018 14715 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.723547 14675 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.724383 14715 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.724776 14715 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.726483 14675 ts_tablet_manager.cc:1428] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:34:24.727883 14675 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.729707 14674 tablet_bootstrap.cc:654] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.734929 14675 tablet_bootstrap.cc:654] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.742726 14679 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.743359 14679 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.746071 14679 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.747151 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.747913 14632 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:24.748129 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.750205 14442 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:24.751674 14679 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.737125 14370 catalog_manager.cc:5495] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:24.761369 14679 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.764303 14683 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.764953 14683 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.759806 14715 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.767058 14683 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.782299 14683 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.782801 14683 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.771703 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.785079 14479 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:24.773801 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.787825 14683 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.789499 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.790323 14479 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:24.791633 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:24.792625 14683 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.793081 14683 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.793622 14683 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.798142 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:34:24.799841 14518 leader_election.cc:341] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.800868 14715 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.806061 14676 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.806049 14675 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:24.806728 14675 ts_tablet_manager.cc:1397] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.079s user 0.034s sys 0.037s
I20260806 20:34:24.806728 14676 ts_tablet_manager.cc:1397] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.092s user 0.021s sys 0.052s
I20260806 20:34:24.806123 14683 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.808211 14674 tablet_bootstrap.cc:492] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:24.809434 14675 raft_consensus.cc:348] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.809664 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:24.809569 14676 raft_consensus.cc:348] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.810175 14675 raft_consensus.cc:374] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.810319 14674 ts_tablet_manager.cc:1397] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.092s user 0.026s sys 0.016s
I20260806 20:34:24.810390 14479 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.810498 14675 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.810412 14676 raft_consensus.cc:374] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.811105 14676 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.811492 14675 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.811879 14676 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.812913 14683 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.814327 14676 ts_tablet_manager.cc:1428] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 20:34:24.815454 14676 tablet_bootstrap.cc:492] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.815807 14707 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.815719 14674 raft_consensus.cc:348] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.816290 14707 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.818436 14707 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:24.819245 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.820255 14674 raft_consensus.cc:374] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.820294 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.820731 14674 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.821085 14555 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:24.822510 14676 tablet_bootstrap.cc:654] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.822400 14596 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:24.823268 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.823905 14681 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.824304 14681 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.824676 14681 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.825453 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.825839 14479 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:24.826056 14632 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.826954 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:24.829243 14521 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:34:24.830435 14521 leader_election.cc:341] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.833467 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:24.835078 14683 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.835702 14683 raft_consensus.cc:686] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:24.835892 14715 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.836647 14683 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.839318 14632 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:24.841125 14707 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 20:34:24.842360 14521 leader_election.cc:341] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:24.841194 14674 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.843619 14715 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:24.844120 14679 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.843636 14521 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:24.844631 14679 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.845542 14674 ts_tablet_manager.cc:1428] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260806 20:34:24.845582 14715 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:24.846449 14715 raft_consensus.cc:686] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:24.846949 14674 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:24.847749 14715 consensus_queue.cc:237] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.848640 14478 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:24.851768 14675 ts_tablet_manager.cc:1428] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.045s user 0.005s sys 0.001s
I20260806 20:34:24.853155 14674 tablet_bootstrap.cc:654] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.853691 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.854420 14555 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:24.855882 14442 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:24.857100 14716 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:24.857560 14716 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:24.857962 14716 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.858681 14745 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.859174 14745 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.861255 14745 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.863947 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:24.866603 14555 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.870472 14675 tablet_bootstrap.cc:492] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:24.874919 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.882329 14679 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.879946 14681 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.886883 14681 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:24.890369 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:24.891384 14554 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:24.894645 14633 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:24.894727 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:24.897748 14675 tablet_bootstrap.cc:654] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:24.900595 14676 tablet_bootstrap.cc:492] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:24.901134 14676 ts_tablet_manager.cc:1397] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.030s
I20260806 20:34:24.898294 14715 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:24.906638 14679 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.907377 14679 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.884133 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.915905 14676 raft_consensus.cc:348] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.917086 14676 raft_consensus.cc:374] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:24.917584 14676 raft_consensus.cc:729] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:24.918249 14676 consensus_queue.cc:260] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.911800 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:24.922078 14716 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.922801 14676 ts_tablet_manager.cc:1428] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260806 20:34:24.923895 14676 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:24.928784 14679 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:24.923187 14554 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:24.936846 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:24.952289 14477 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:24.949999 14683 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.962000 14707 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.993039 14683 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.993254 14707 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:24.994968 14370 catalog_manager.cc:5495] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:24.970763 14596 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:24.955940 14745 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:24.972934 14633 raft_consensus.cc:2379] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:24.975338 14715 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 20:34:24.964792 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:25.012283 14675 tablet_bootstrap.cc:492] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:25.019829 14675 ts_tablet_manager.cc:1397] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.150s user 0.026s sys 0.004s
I20260806 20:34:24.995873 14707 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:34:25.021574 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 50962.6us
I20260806 20:34:25.024744 14707 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.031955 14753 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.028992 14683 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.032560 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.033067 14633 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.032577 14716 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.041766 14756 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:25.034410 14444 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.040596 14753 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.040220 14707 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.045644 14753 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.045619 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:25.046146 14707 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.035522 14674 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:25.019146 14745 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.026204 14521 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.037025 14755 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.038602 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:25.040673 14477 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:25.031347 14681 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:25.034516 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.048441 14674 ts_tablet_manager.cc:1397] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.202s user 0.031s sys 0.064s
I20260806 20:34:25.055310 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.054224 14755 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.055976 14554 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:25.056572 14553 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:25.057247 14755 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.058341 14518 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:25.060601 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.051024 14728 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.039777 14679 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 20:34:25.078750 14745 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.072602 14676 tablet_bootstrap.cc:654] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.065333 14675 raft_consensus.cc:348] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.077769 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.080509 14675 raft_consensus.cc:374] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.081889 14675 raft_consensus.cc:729] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.082281 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:25.083072 14675 consensus_queue.cc:260] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.082948 14632 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:25.084968 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.082281 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.085675 14675 ts_tablet_manager.cc:1428] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.065s user 0.009s sys 0.000s
I20260806 20:34:25.085842 14476 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:25.086184 14555 raft_consensus.cc:2379] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:25.064083 14683 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.086555 14675 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:25.089533 14674 raft_consensus.cc:348] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.090495 14674 raft_consensus.cc:374] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.052310 14716 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260806 20:34:25.096026 14763 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:25.062853 14681 raft_consensus.cc:686] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:25.099133 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.090791 14674 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.109835 14674 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.103515 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:25.113133 14674 ts_tablet_manager.cc:1428] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.059s user 0.007s sys 0.002s
W20260806 20:34:25.091749 14595 leader_election.cc:341] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.116868 14674 tablet_bootstrap.cc:492] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:25.115265 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:25.117731 14595 leader_election.cc:341] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.114605 14681 consensus_queue.cc:237] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.026157 14370 catalog_manager.cc:5495] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
W20260806 20:34:25.118256 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 60012us
I20260806 20:34:25.100376 14683 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:25.125010 14596 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:34:25.125185 14521 leader_election.cc:341] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.120820 14595 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:25.130857 14683 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:25.148101 14756 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:25.148690 14756 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:25.149123 14756 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.141220 14683 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.178020 14761 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260806 20:34:25.175145 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 77250.8us
I20260806 20:34:25.181846 14596 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:34:25.183020 14596 leader_election.cc:341] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.183441 14596 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:25.184140 14753 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.143041 14728 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.143599 14707 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:25.184763 14753 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.189865 14676 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:25.190496 14676 ts_tablet_manager.cc:1397] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.267s user 0.024s sys 0.016s
I20260806 20:34:25.186543 14707 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:25.194046 14707 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.192642 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.202633 14753 consensus_queue.cc:1035] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260806 20:34:25.209380 14755 consensus_queue.cc:1035] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:25.211148 14763 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:25.191080 14728 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20260806 20:34:25.228677 14518 leader_election.cc:341] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.230619 14518 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.233054 14760 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 20:34:25.243178 14521 consensus_peers.cc:487] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Couldn't send request to peer df1245ecd2034e11ae6cf688c1c10a4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:25.336984 14675 tablet_bootstrap.cc:654] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.250816 14676 raft_consensus.cc:348] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.334892 14745 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 20:34:25.338357 14676 raft_consensus.cc:374] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.338665 14676 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
W20260806 20:34:25.336516 14518 consensus_peers.cc:487] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Couldn't send request to peer 92828dd6dbe74f29a04d4e702b502621. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:25.336097 14370 catalog_manager.cc:5495] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:25.343744 14674 tablet_bootstrap.cc:654] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.345466 14676 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.316679 14728 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260806 20:34:25.330263 14683 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.299171 14707 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.362835 14707 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.363014 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:25.363617 14476 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.363917 14553 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:25.364602 14683 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.366287 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:25.366882 14477 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.294956 14756 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.372303 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:25.357192 14676 ts_tablet_manager.cc:1428] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.166s user 0.013s sys 0.009s
I20260806 20:34:25.377830 14676 tablet_bootstrap.cc:492] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:25.260401 14444 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.264249 14760 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.112s
I20260806 20:34:25.381201 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:25.480253 14632 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:25.386561 14477 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:25.378973 14554 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.410517 14476 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:25.440353 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:25.482021 14479 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:25.482424 14518 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:25.440629 14756 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.443285 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:25.444005 14707 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.484230 14707 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.480280 14679 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.463238 14675 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:25.466960 14683 consensus_queue.cc:1035] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:25.487430 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.488154 14675 ts_tablet_manager.cc:1397] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.402s user 0.089s sys 0.075s
W20260806 20:34:25.488691 14596 leader_election.cc:341] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.488729 14554 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:25.489337 14707 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.491438 14675 raft_consensus.cc:348] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.491884 14675 raft_consensus.cc:374] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.492028 14675 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.492619 14675 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.471727 14676 tablet_bootstrap.cc:654] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.475195 14674 tablet_bootstrap.cc:492] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:25.496809 14674 ts_tablet_manager.cc:1397] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.380s user 0.015s sys 0.006s
W20260806 20:34:25.480679 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 220400us
I20260806 20:34:25.483366 14761 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:25.498214 14761 raft_consensus.cc:686] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:25.499660 14674 raft_consensus.cc:348] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.483378 14595 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:25.500633 14674 raft_consensus.cc:374] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.501004 14674 raft_consensus.cc:729] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.502702 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.503744 14675 ts_tablet_manager.cc:1428] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260806 20:34:25.503852 14756 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:25.504662 14756 raft_consensus.cc:686] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:25.505043 14675 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:25.505839 14756 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.499382 14761 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.501814 14674 consensus_queue.cc:260] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.512538 14476 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:25.514091 14674 ts_tablet_manager.cc:1428] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260806 20:34:25.514786 14595 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:25.516366 14707 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:25.516831 14707 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:25.517238 14707 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.521523 14675 tablet_bootstrap.cc:654] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.515278 14674 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:25.564249 14676 tablet_bootstrap.cc:492] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:25.564852 14676 ts_tablet_manager.cc:1397] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.187s user 0.018s sys 0.032s
I20260806 20:34:25.586934 14674 tablet_bootstrap.cc:654] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.580658 14596 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:34:25.622710 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 45725.3us
I20260806 20:34:25.623306 14755 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.623710 14753 consensus_queue.cc:1035] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:25.632442 14707 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.640637 14676 raft_consensus.cc:348] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.641786 14676 raft_consensus.cc:374] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.642112 14676 raft_consensus.cc:729] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.643070 14676 consensus_queue.cc:260] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.643465 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:25.644236 14477 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.645665 14676 ts_tablet_manager.cc:1428] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.080s user 0.006s sys 0.002s
I20260806 20:34:25.646997 14676 tablet_bootstrap.cc:492] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:25.659878 14753 consensus_queue.cc:1035] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:25.669075 14707 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.671810 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:25.674013 14477 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:25.695593 14675 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:25.696110 14675 ts_tablet_manager.cc:1397] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.191s user 0.020s sys 0.000s
W20260806 20:34:25.675104 14596 leader_election.cc:341] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.701071 14371 catalog_manager.cc:5495] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:25.702603 14595 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:25.704686 14707 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:25.705237 14707 raft_consensus.cc:686] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:25.705585 14674 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:25.706049 14674 ts_tablet_manager.cc:1397] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.191s user 0.008s sys 0.015s
I20260806 20:34:25.706115 14707 consensus_queue.cc:237] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.712574 14676 tablet_bootstrap.cc:654] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.715776 14674 raft_consensus.cc:348] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.716590 14674 raft_consensus.cc:374] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.716871 14674 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.717620 14674 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.723307 14675 raft_consensus.cc:348] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.724084 14675 raft_consensus.cc:374] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.724373 14675 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.725090 14675 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.727324 14675 ts_tablet_manager.cc:1428] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260806 20:34:25.728240 14675 tablet_bootstrap.cc:492] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:25.736243 14674 ts_tablet_manager.cc:1428] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260806 20:34:25.737332 14674 tablet_bootstrap.cc:492] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:25.741709 14632 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.749562 14676 tablet_bootstrap.cc:492] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:25.750068 14676 ts_tablet_manager.cc:1397] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.001s
I20260806 20:34:25.761420 14674 tablet_bootstrap.cc:654] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.770650 14372 catalog_manager.cc:5495] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:25.772291 14675 tablet_bootstrap.cc:654] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.773564 14676 raft_consensus.cc:348] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.775993 14676 raft_consensus.cc:374] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.776335 14676 raft_consensus.cc:729] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.784634 14676 consensus_queue.cc:260] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.787416 14674 tablet_bootstrap.cc:492] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:25.787866 14674 ts_tablet_manager.cc:1397] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.002s
I20260806 20:34:25.790820 14674 raft_consensus.cc:348] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.791587 14674 raft_consensus.cc:374] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.791893 14674 raft_consensus.cc:729] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.792603 14674 consensus_queue.cc:260] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.795006 14674 ts_tablet_manager.cc:1428] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 20:34:25.796020 14674 tablet_bootstrap.cc:492] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:25.798204 14784 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.798825 14784 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.802816 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.803354 14784 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.805178 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:25.806835 14442 leader_election.cc:341] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:25.807997 14444 leader_election.cc:341] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.808466 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.810072 14676 ts_tablet_manager.cc:1428] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.060s user 0.007s sys 0.000s
I20260806 20:34:25.810817 14745 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.811127 14676 tablet_bootstrap.cc:492] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:25.811300 14745 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.818215 14784 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.818883 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.819584 14707 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:34:25.820290 14518 leader_election.cc:341] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.820966 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:25.821595 14630 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:25.822747 14675 tablet_bootstrap.cc:492] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:25.823222 14675 ts_tablet_manager.cc:1397] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.008s
I20260806 20:34:25.820111 14707 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.818949 14745 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.827713 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.831683 14675 raft_consensus.cc:348] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.832489 14675 raft_consensus.cc:374] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.832789 14675 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.827569 14521 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:25.833496 14675 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.833973 14683 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:25.834362 14683 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:25.834923 14683 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.829941 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.835887 14675 ts_tablet_manager.cc:1428] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260806 20:34:25.828369 14371 catalog_manager.cc:5495] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:25.828828 14707 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.839998 14674 tablet_bootstrap.cc:654] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:25.833513 14595 leader_election.cc:341] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:25.840730 14596 leader_election.cc:341] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.841214 14683 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.841599 14596 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:25.842788 14707 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.844448 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:34:25.845952 14518 leader_election.cc:341] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.846622 14676 tablet_bootstrap.cc:654] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.846977 14683 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.848349 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:25.849032 14630 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.850306 14675 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:25.856294 14675 tablet_bootstrap.cc:654] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.859018 14630 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:25.860086 14521 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:25.860984 14683 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:25.861454 14683 raft_consensus.cc:686] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:25.862305 14683 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.860296 14745 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.864850 14745 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.867211 14745 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.868917 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:25.869856 14676 tablet_bootstrap.cc:492] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:25.869323 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.870502 14676 ts_tablet_manager.cc:1397] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.008s
W20260806 20:34:25.870525 14521 leader_election.cc:341] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:25.871537 14518 leader_election.cc:341] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.871961 14518 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.873994 14745 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.875682 14676 raft_consensus.cc:348] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.876689 14676 raft_consensus.cc:374] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.877113 14676 raft_consensus.cc:729] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.878363 14674 tablet_bootstrap.cc:492] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:25.877959 14676 consensus_queue.cc:260] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.878985 14674 ts_tablet_manager.cc:1397] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.083s user 0.010s sys 0.009s
I20260806 20:34:25.879876 14745 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.880398 14745 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.880904 14676 ts_tablet_manager.cc:1428] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20260806 20:34:25.882689 14676 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:25.883195 14675 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:25.883366 14707 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.883666 14675 ts_tablet_manager.cc:1397] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.001s
I20260806 20:34:25.883872 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:25.883849 14707 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:25.885368 14521 leader_election.cc:341] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.886817 14707 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.888852 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.889077 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.889897 14676 tablet_bootstrap.cc:654] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:25.890265 14596 leader_election.cc:341] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:25.890710 14595 leader_election.cc:341] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.891227 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:25.892673 14707 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:25.894083 14745 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.894596 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.895279 14476 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:25.896554 14518 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.897687 14745 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:25.898039 14745 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:25.898371 14745 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.905117 14676 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:25.904517 14372 catalog_manager.cc:5495] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:25.905570 14676 ts_tablet_manager.cc:1397] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.004s
I20260806 20:34:25.907060 14745 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.909898 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:25.910503 14476 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.911401 14675 raft_consensus.cc:348] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.912139 14675 raft_consensus.cc:374] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.912405 14675 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.914543 14674 raft_consensus.cc:348] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.919885 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:25.920244 14674 raft_consensus.cc:374] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.920601 14674 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
W20260806 20:34:25.921453 14521 leader_election.cc:341] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.921306 14674 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.916227 14676 raft_consensus.cc:348] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.923535 14676 raft_consensus.cc:374] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:25.917898 14476 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:25.924888 14518 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.918743 14745 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.926208 14674 ts_tablet_manager.cc:1428] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260806 20:34:25.926584 14745 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:25.927170 14674 tablet_bootstrap.cc:492] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:25.927912 14745 raft_consensus.cc:686] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:25.928872 14745 consensus_queue.cc:237] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.930295 14707 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:25.933704 14674 tablet_bootstrap.cc:654] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.919377 14675 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.923835 14676 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:25.935668 14676 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.944499 14676 ts_tablet_manager.cc:1428] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.039s user 0.007s sys 0.001s
I20260806 20:34:25.948598 14676 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:25.950551 14753 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:25.952678 14371 catalog_manager.cc:5495] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:25.965340 14675 ts_tablet_manager.cc:1428] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.081s user 0.010s sys 0.000s
I20260806 20:34:25.966663 14675 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:25.971359 14676 tablet_bootstrap.cc:654] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.971946 14753 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.972523 14753 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.974776 14753 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:25.977562 14745 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:25.978188 14745 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:25.979178 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:25.980367 14745 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:25.979183 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.981225 14477 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:25.983057 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:25.983820 14479 raft_consensus.cc:2379] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:25.984759 14595 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:25.985838 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:25.988197 14596 leader_election.cc:341] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:25.989436 14707 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:25.989768 14707 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:25.990087 14707 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:25.993957 14675 tablet_bootstrap.cc:654] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:25.995738 14518 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:25.996613 14745 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.004158 14683 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:26.009644 14745 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.020556 14745 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.029469 14707 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.039126 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:34:26.040518 14596 leader_election.cc:341] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.041167 14707 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.041999 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.042649 14476 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.083448 14674 tablet_bootstrap.cc:492] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:26.089876 14674 ts_tablet_manager.cc:1397] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.163s user 0.034s sys 0.046s
I20260806 20:34:26.085742 14476 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
W20260806 20:34:26.073477 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:26.092869 14674 raft_consensus.cc:348] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.094333 14676 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.095721 14676 ts_tablet_manager.cc:1397] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.148s user 0.020s sys 0.023s
I20260806 20:34:26.096437 14674 raft_consensus.cc:374] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.096774 14674 raft_consensus.cc:729] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.097273 14674 consensus_queue.cc:260] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.096503 14595 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:26.099454 14707 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.100124 14707 raft_consensus.cc:686] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:26.099956 14676 raft_consensus.cc:348] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.100709 14676 raft_consensus.cc:374] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.100960 14676 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.100994 14707 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.101678 14676 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.103962 14676 ts_tablet_manager.cc:1428] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260806 20:34:26.105852 14676 tablet_bootstrap.cc:492] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.111563 14676 tablet_bootstrap.cc:654] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.105875 14674 ts_tablet_manager.cc:1428] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260806 20:34:26.113773 14675 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.114285 14675 ts_tablet_manager.cc:1397] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.148s user 0.013s sys 0.018s
I20260806 20:34:26.114329 14674 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:26.128158 14370 catalog_manager.cc:5495] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:26.129000 14675 raft_consensus.cc:348] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.130528 14675 raft_consensus.cc:374] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.130862 14675 raft_consensus.cc:729] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.131834 14675 consensus_queue.cc:260] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.135013 14675 ts_tablet_manager.cc:1428] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260806 20:34:26.136123 14675 tablet_bootstrap.cc:492] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.142354 14674 tablet_bootstrap.cc:654] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.149073 14675 tablet_bootstrap.cc:654] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.144505 14676 tablet_bootstrap.cc:492] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.151633 14676 ts_tablet_manager.cc:1397] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.021s
I20260806 20:34:26.153417 14476 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.154138 14676 raft_consensus.cc:348] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.154807 14676 raft_consensus.cc:374] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.155076 14676 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.155762 14676 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.157793 14676 ts_tablet_manager.cc:1428] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:34:26.158635 14676 tablet_bootstrap.cc:492] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.161283 14707 consensus_queue.cc:1035] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:26.163559 14674 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:26.164139 14674 ts_tablet_manager.cc:1397] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.003s
I20260806 20:34:26.164685 14676 tablet_bootstrap.cc:654] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.170657 14674 raft_consensus.cc:348] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.174314 14674 raft_consensus.cc:374] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.174620 14674 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.175297 14674 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.171523 14675 tablet_bootstrap.cc:492] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.176793 14675 ts_tablet_manager.cc:1397] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.016s
I20260806 20:34:26.190755 14674 ts_tablet_manager.cc:1428] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.026s user 0.008s sys 0.001s
I20260806 20:34:26.191722 14674 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:26.199245 14675 raft_consensus.cc:348] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.202811 14675 raft_consensus.cc:374] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.203233 14675 raft_consensus.cc:729] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.204288 14675 consensus_queue.cc:260] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.207880 14675 ts_tablet_manager.cc:1428] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20260806 20:34:26.210038 14675 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.213593 14753 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.214073 14753 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.214910 14676 tablet_bootstrap.cc:492] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.215327 14676 ts_tablet_manager.cc:1397] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.033s
I20260806 20:34:26.215962 14753 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.217757 14676 raft_consensus.cc:348] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.218453 14676 raft_consensus.cc:374] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.218266 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:26.219821 14596 leader_election.cc:341] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.220400 14753 consensus_queue.cc:1035] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.220825 14676 raft_consensus.cc:729] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.220760 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.221648 14676 consensus_queue.cc:260] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:26.223642 14595 leader_election.cc:341] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.223897 14676 ts_tablet_manager.cc:1428] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260806 20:34:26.224090 14595 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:26.224845 14676 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.225643 14755 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.226787 14675 tablet_bootstrap.cc:654] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.235327 14676 tablet_bootstrap.cc:654] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.248980 14675 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.249549 14675 ts_tablet_manager.cc:1397] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.007s
I20260806 20:34:26.252174 14675 raft_consensus.cc:348] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.253007 14675 raft_consensus.cc:374] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.255079 14675 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.257562 14675 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.259902 14675 ts_tablet_manager.cc:1428] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 20:34:26.259925 14683 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.260519 14683 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.261024 14675 tablet_bootstrap.cc:492] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.263017 14683 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.263377 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.264043 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:26.264688 14518 leader_election.cc:341] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:26.265584 14521 leader_election.cc:341] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.266026 14521 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:26.267203 14683 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.279453 14755 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.279978 14755 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.282228 14755 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.282829 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.283514 14476 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:26.283694 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:26.284361 14554 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:26.284647 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.285840 14755 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:26.286170 14755 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.286484 14755 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.286866 14676 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.287312 14676 ts_tablet_manager.cc:1397] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.012s
I20260806 20:34:26.289742 14676 raft_consensus.cc:348] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.290400 14676 raft_consensus.cc:374] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.290652 14676 raft_consensus.cc:729] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.291339 14676 consensus_queue.cc:260] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.293399 14676 ts_tablet_manager.cc:1428] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 20:34:26.293810 14674 tablet_bootstrap.cc:654] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.296423 14676 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.297653 14755 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.300791 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.301405 14477 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.302747 14676 tablet_bootstrap.cc:654] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.304620 14675 tablet_bootstrap.cc:654] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.309675 14755 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.308169 14477 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:26.311371 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.311506 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:26.312105 14554 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.312201 14755 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.313928 14674 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:26.314424 14674 ts_tablet_manager.cc:1397] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.002s
I20260806 20:34:26.314832 14755 raft_consensus.cc:686] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:26.316152 14755 consensus_queue.cc:237] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.317060 14761 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.317492 14674 raft_consensus.cc:348] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.317495 14761 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.318471 14674 raft_consensus.cc:374] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.318801 14674 raft_consensus.cc:729] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.319759 14761 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.318109 14683 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.320708 14675 tablet_bootstrap.cc:492] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.320309 14683 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.321349 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.321700 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:26.322225 14683 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.322356 14630 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:26.322654 14554 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:26.322877 14675 ts_tablet_manager.cc:1397] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.002s
W20260806 20:34:26.323621 14518 leader_election.cc:341] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.325209 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.325290 14521 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:26.326380 14683 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:26.326472 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:26.326781 14683 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.319447 14674 consensus_queue.cc:260] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.327158 14683 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.329092 14674 ts_tablet_manager.cc:1428] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 20:34:26.327124 14630 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:26.330284 14761 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.330664 14675 raft_consensus.cc:348] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.331350 14675 raft_consensus.cc:374] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.330941 14761 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.331631 14675 raft_consensus.cc:729] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.332410 14675 consensus_queue.cc:260] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.332157 14369 catalog_manager.cc:5495] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:26.333962 14521 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.334096 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.334718 14476 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:26.335493 14745 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:26.335829 14745 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.336539 14745 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.341866 14674 tablet_bootstrap.cc:492] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:26.342917 14518 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:26.341832 14683 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.348562 14675 ts_tablet_manager.cc:1428] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260806 20:34:26.349848 14675 tablet_bootstrap.cc:492] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.357264 14675 tablet_bootstrap.cc:654] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.360673 14683 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.363967 14683 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:26.367647 14683 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.368098 14683 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.368732 14755 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.369170 14755 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.371136 14755 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.364480 14761 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.373016 14676 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.373473 14676 ts_tablet_manager.cc:1397] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.000s
I20260806 20:34:26.373324 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:26.374754 14595 leader_election.cc:341] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.374778 14674 tablet_bootstrap.cc:654] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.376015 14676 raft_consensus.cc:348] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.376796 14676 raft_consensus.cc:374] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.377173 14676 raft_consensus.cc:729] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.377709 14676 consensus_queue.cc:260] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:26.365000 14518 leader_election.cc:341] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.380924 14676 ts_tablet_manager.cc:1428] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:34:26.380776 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:26.380544 14745 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.381804 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:26.384701 14630 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.384646 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.386829 14683 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.387336 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:26.388149 14632 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:26.388959 14683 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.389262 14745 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.391242 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:26.392042 14632 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.392264 14630 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:26.393783 14676 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
W20260806 20:34:26.396895 14596 leader_election.cc:341] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.397348 14596 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:26.398062 14755 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.399778 14675 tablet_bootstrap.cc:492] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.400207 14675 ts_tablet_manager.cc:1397] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.051s user 0.016s sys 0.000s
I20260806 20:34:26.400823 14676 tablet_bootstrap.cc:654] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.402616 14675 raft_consensus.cc:348] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.403321 14675 raft_consensus.cc:374] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.403266 14632 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:26.403707 14675 raft_consensus.cc:729] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.405616 14676 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.406124 14676 ts_tablet_manager.cc:1397] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 20:34:26.406299 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.408933 14479 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.408977 14521 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.409890 14521 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.410308 14745 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:34:26.410624 14518 leader_election.cc:341] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.411716 14745 raft_consensus.cc:686] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:26.412602 14745 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.411731 14683 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.416020 14683 raft_consensus.cc:686] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:26.406836 14476 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.418174 14828 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.415534 14479 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:26.406699 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.423971 14676 raft_consensus.cc:348] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.424700 14676 raft_consensus.cc:374] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.425011 14676 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.407692 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:26.425595 14630 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.425673 14676 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.426221 14828 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.427775 14676 ts_tablet_manager.cc:1428] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
W20260806 20:34:26.428094 14518 leader_election.cc:341] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.428642 14676 tablet_bootstrap.cc:492] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.434526 14630 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:26.404381 14675 consensus_queue.cc:260] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.417130 14683 consensus_queue.cc:237] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.437075 14521 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:26.437885 14745 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.438535 14745 raft_consensus.cc:686] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:26.407696 14674 tablet_bootstrap.cc:492] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:26.439770 14675 ts_tablet_manager.cc:1428] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260806 20:34:26.439492 14745 consensus_queue.cc:237] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.441109 14675 tablet_bootstrap.cc:492] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.442540 14674 ts_tablet_manager.cc:1397] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.101s user 0.014s sys 0.006s
I20260806 20:34:26.447434 14683 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.447881 14683 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.449455 14674 raft_consensus.cc:348] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.450951 14674 raft_consensus.cc:374] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.451315 14674 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.451812 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.452973 14478 raft_consensus.cc:2379] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:26.453115 14683 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.453944 14674 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.457849 14674 ts_tablet_manager.cc:1428] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260806 20:34:26.458933 14674 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:26.469300 14676 tablet_bootstrap.cc:654] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.509338 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:26.513952 14370 catalog_manager.cc:5495] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:26.515414 14370 catalog_manager.cc:5495] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:26.520592 14683 consensus_queue.cc:1035] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.521760 14675 tablet_bootstrap.cc:654] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.528772 14521 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:26.535255 14828 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.546373 14674 tablet_bootstrap.cc:654] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.552073 14628 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.559728 14707 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.560586 14828 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.566848 14675 tablet_bootstrap.cc:492] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.567569 14675 ts_tablet_manager.cc:1397] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.127s user 0.016s sys 0.001s
I20260806 20:34:26.571998 14369 catalog_manager.cc:5495] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:26.577360 14784 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.577850 14784 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.580030 14784 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.582780 14675 raft_consensus.cc:348] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.583573 14675 raft_consensus.cc:374] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.582178 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:26.583873 14675 raft_consensus.cc:729] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.584051 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:26.584403 14632 raft_consensus.cc:2379] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:26.584656 14675 consensus_queue.cc:260] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.588074 14828 consensus_queue.cc:1035] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.590444 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:26.592643 14784 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.593184 14675 ts_tablet_manager.cc:1428] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20260806 20:34:26.594019 14675 tablet_bootstrap.cc:492] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.595213 14676 tablet_bootstrap.cc:492] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.595635 14676 ts_tablet_manager.cc:1397] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.167s user 0.022s sys 0.023s
I20260806 20:34:26.596292 14707 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 20:34:26.611238 14674 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:26.611645 14674 ts_tablet_manager.cc:1397] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.153s user 0.009s sys 0.006s
I20260806 20:34:26.613493 14676 raft_consensus.cc:348] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.614225 14676 raft_consensus.cc:374] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.614493 14676 raft_consensus.cc:729] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.615217 14676 consensus_queue.cc:260] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.615981 14674 raft_consensus.cc:348] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.616686 14674 raft_consensus.cc:374] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.616940 14674 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.617659 14676 ts_tablet_manager.cc:1428] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260806 20:34:26.617558 14674 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.618752 14676 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.622663 14675 tablet_bootstrap.cc:654] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.632591 14784 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.635406 14756 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.636415 14674 ts_tablet_manager.cc:1428] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.024s user 0.005s sys 0.005s
I20260806 20:34:26.637405 14674 tablet_bootstrap.cc:492] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:26.641026 14676 tablet_bootstrap.cc:654] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.635941 14756 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.642784 14674 tablet_bootstrap.cc:654] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.645789 14756 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.646286 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.647624 14675 tablet_bootstrap.cc:492] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.648061 14675 ts_tablet_manager.cc:1397] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.003s
I20260806 20:34:26.648864 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:26.649479 14554 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:26.650427 14596 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.650485 14675 raft_consensus.cc:348] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.651208 14675 raft_consensus.cc:374] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.651286 14756 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:26.651504 14675 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.651645 14756 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.652125 14756 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.652220 14675 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:26.653066 14595 leader_election.cc:341] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.654299 14675 ts_tablet_manager.cc:1428] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 20:34:26.655098 14675 tablet_bootstrap.cc:492] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.664340 14756 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.667682 14676 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.668223 14676 ts_tablet_manager.cc:1397] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.004s
I20260806 20:34:26.669265 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.669801 14675 tablet_bootstrap.cc:654] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:26.670583 14595 leader_election.cc:341] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.671042 14676 raft_consensus.cc:348] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.671700 14676 raft_consensus.cc:374] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.671973 14676 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.672621 14676 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.674063 14674 tablet_bootstrap.cc:492] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:26.674612 14674 ts_tablet_manager.cc:1397] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.010s
I20260806 20:34:26.678392 14674 raft_consensus.cc:348] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.679220 14674 raft_consensus.cc:374] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.679538 14674 raft_consensus.cc:729] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.680559 14676 ts_tablet_manager.cc:1428] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 20:34:26.681468 14676 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.683532 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:26.684105 14554 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.691593 14841 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.692238 14841 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.693156 14676 tablet_bootstrap.cc:654] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.696089 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:26.696827 14632 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:26.697968 14444 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.698530 14841 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.699151 14674 consensus_queue.cc:260] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.698660 14675 tablet_bootstrap.cc:492] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.700541 14675 ts_tablet_manager.cc:1397] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.012s
I20260806 20:34:26.698026 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:26.701181 14553 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:26.701992 14676 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.702476 14676 ts_tablet_manager.cc:1397] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20260806 20:34:26.698946 14784 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:26.702875 14784 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.703243 14784 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.703049 14675 raft_consensus.cc:348] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.703399 14756 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.703722 14675 raft_consensus.cc:374] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.704007 14675 raft_consensus.cc:729] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.704618 14675 consensus_queue.cc:260] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.704805 14676 raft_consensus.cc:348] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.705494 14676 raft_consensus.cc:374] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.705741 14676 raft_consensus.cc:729] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.706650 14675 ts_tablet_manager.cc:1428] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:34:26.706321 14676 consensus_queue.cc:260] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.709437 14745 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.709928 14745 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.711957 14745 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.711993 14784 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.713121 14707 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.713564 14707 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.715092 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:26.715390 14707 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.715651 14553 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.717216 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.718255 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:26.718465 14595 leader_election.cc:341] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.718832 14476 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:26.720155 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:26.722167 14518 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:26.723318 14761 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 20:34:26.723402 14596 leader_election.cc:341] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.723703 14761 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.724215 14596 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:26.724373 14784 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.723397 14674 ts_tablet_manager.cc:1428] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260806 20:34:26.724287 14761 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.725373 14707 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.727063 14676 ts_tablet_manager.cc:1428] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260806 20:34:26.727849 14676 tablet_bootstrap.cc:492] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.729336 14674 tablet_bootstrap.cc:492] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:26.731618 14761 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.733525 14554 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:26.734455 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.734548 14596 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:26.735054 14476 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.735625 14745 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.736217 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:26.736690 14675 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
W20260806 20:34:26.737504 14521 leader_election.cc:341] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.735766 14756 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.738667 14756 raft_consensus.cc:686] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:26.736169 14745 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.740643 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:26.740942 14745 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.741238 14628 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.742296 14675 tablet_bootstrap.cc:654] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.742391 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:26.744273 14521 leader_election.cc:341] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.735684 14674 tablet_bootstrap.cc:654] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.745051 14707 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.745609 14756 consensus_queue.cc:237] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.747493 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:26.749465 14518 leader_election.cc:341] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.750020 14518 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:26.751058 14745 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.745653 14707 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.757349 14676 tablet_bootstrap.cc:654] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.760274 14707 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.763720 14476 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:26.764631 14628 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:26.768064 14444 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.768311 14761 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.769089 14784 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.769552 14784 raft_consensus.cc:686] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:26.769413 14518 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:26.770423 14784 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.771203 14761 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.771628 14761 raft_consensus.cc:686] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:26.772397 14761 consensus_queue.cc:237] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.770341 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:34:26.776526 14521 leader_election.cc:341] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.758464 14553 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:26.777490 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:26.789069 14554 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:26.761384 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.792485 14596 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.795516 14707 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:26.795879 14707 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:26.796190 14707 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.801294 14372 catalog_manager.cc:5495] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:26.823666 14370 catalog_manager.cc:5495] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:26.824436 14369 catalog_manager.cc:5495] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:26.792339 14478 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:26.840034 14756 consensus_queue.cc:1035] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:26.844100 14707 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.847240 14707 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:26.848403 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:26.849242 14553 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.869849 14761 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.870384 14761 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.900543 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:26.902235 14518 leader_election.cc:341] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.904824 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:26.906226 14521 leader_election.cc:341] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:26.906687 14521 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:26.907637 14745 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:26.908337 14675 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:26.908653 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:26.908803 14675 ts_tablet_manager.cc:1397] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.172s user 0.020s sys 0.020s
I20260806 20:34:26.909040 14676 tablet_bootstrap.cc:492] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:26.909227 14478 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:26.916656 14676 ts_tablet_manager.cc:1397] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.189s user 0.033s sys 0.010s
I20260806 20:34:26.919683 14676 raft_consensus.cc:348] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.909394 14674 tablet_bootstrap.cc:492] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:26.920542 14676 raft_consensus.cc:374] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.920773 14674 ts_tablet_manager.cc:1397] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.192s user 0.012s sys 0.057s
I20260806 20:34:26.920907 14676 raft_consensus.cc:729] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.921727 14676 consensus_queue.cc:260] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.923321 14674 raft_consensus.cc:348] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.923988 14674 raft_consensus.cc:374] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.924357 14676 ts_tablet_manager.cc:1428] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:34:26.924495 14674 raft_consensus.cc:729] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.925348 14674 consensus_queue.cc:260] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.927349 14674 ts_tablet_manager.cc:1428] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:34:26.927999 14478 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:26.913246 14553 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:26.913268 14828 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260806 20:34:26.929970 14596 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:26.930423 14826 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:26.910022 14761 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.931990 14674 tablet_bootstrap.cc:492] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:26.913415 14745 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260806 20:34:26.937238 14707 consensus_queue.cc:1035] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.938103 14674 tablet_bootstrap.cc:654] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.930780 14756 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:26.945832 14756 raft_consensus.cc:686] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:34:26.946283 14518 consensus_peers.cc:487] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Couldn't send request to peer 92828dd6dbe74f29a04d4e702b502621. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:26.946767 14756 consensus_queue.cc:237] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.953637 14745 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:26.957916 14675 raft_consensus.cc:348] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.958602 14675 raft_consensus.cc:374] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:26.958868 14675 raft_consensus.cc:729] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:26.959611 14675 consensus_queue.cc:260] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.960799 14676 tablet_bootstrap.cc:492] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:26.966998 14675 ts_tablet_manager.cc:1428] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20260806 20:34:26.979602 14675 tablet_bootstrap.cc:492] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:26.985414 14675 tablet_bootstrap.cc:654] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.990181 14676 tablet_bootstrap.cc:654] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:26.992926 14761 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:26.993538 14761 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:26.996856 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:26.998208 14761 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:26.999104 14676 tablet_bootstrap.cc:492] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
W20260806 20:34:26.999230 14518 leader_election.cc:341] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.000782 14676 ts_tablet_manager.cc:1397] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.001s
I20260806 20:34:27.001295 14674 tablet_bootstrap.cc:492] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.001807 14674 ts_tablet_manager.cc:1397] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.016s
I20260806 20:34:27.003921 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.004638 14628 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:27.007918 14674 raft_consensus.cc:348] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.007864 14521 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.008775 14707 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.009404 14707 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.010502 14675 tablet_bootstrap.cc:492] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:27.010623 14827 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.010437 14676 raft_consensus.cc:348] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.011020 14827 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.011210 14675 ts_tablet_manager.cc:1397] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.003s
I20260806 20:34:27.011376 14676 raft_consensus.cc:374] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.011485 14827 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.011705 14676 raft_consensus.cc:729] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.012682 14707 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.012374 14676 consensus_queue.cc:260] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.013600 14675 raft_consensus.cc:348] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.014283 14675 raft_consensus.cc:374] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.014516 14676 ts_tablet_manager.cc:1428] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260806 20:34:27.016124 14676 tablet_bootstrap.cc:492] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.018864 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:27.020632 14595 leader_election.cc:341] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.021485 14676 tablet_bootstrap.cc:654] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.014537 14675 raft_consensus.cc:729] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.029372 14370 catalog_manager.cc:5495] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:27.029376 14675 consensus_queue.cc:260] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.008775 14674 raft_consensus.cc:374] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.032023 14674 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.033145 14675 ts_tablet_manager.cc:1428] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260806 20:34:27.034039 14675 tablet_bootstrap.cc:492] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:27.034127 14827 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.033717 14674 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.039686 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:34:27.041941 14518 leader_election.cc:341] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.035218 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.043844 14554 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:27.044970 14596 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.046603 14756 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.046980 14756 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.047331 14756 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.047735 14827 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.048671 14674 ts_tablet_manager.cc:1428] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.047s user 0.008s sys 0.000s
I20260806 20:34:27.048903 14676 tablet_bootstrap.cc:492] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.051219 14826 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:27.057098 14674 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.060124 14676 ts_tablet_manager.cc:1397] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.044s user 0.025s sys 0.007s
I20260806 20:34:27.053849 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:27.060633 14628 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.054332 14756 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.061756 14784 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.062188 14784 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.064077 14784 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.065876 14756 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.065932 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.066689 14630 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:27.067214 14628 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:27.067463 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.069550 14444 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:34:27.070354 14442 leader_election.cc:341] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.071187 14826 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.071661 14826 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.071957 14521 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.071686 14784 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.072762 14745 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.072908 14784 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.073285 14745 raft_consensus.cc:686] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:27.073294 14784 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.073688 14826 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.074118 14745 consensus_queue.cc:237] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.074507 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.074723 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:27.075284 14554 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:27.075888 14518 leader_election.cc:341] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.077471 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:27.078600 14521 leader_election.cc:341] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.079010 14521 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.079715 14827 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.051563 14675 tablet_bootstrap.cc:654] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.084416 14554 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:27.086799 14596 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:27.087796 14756 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.088323 14756 raft_consensus.cc:686] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:27.089744 14674 tablet_bootstrap.cc:654] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.070698 14676 raft_consensus.cc:348] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.090162 14676 raft_consensus.cc:374] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.090495 14676 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.085640 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:27.089242 14756 consensus_queue.cc:237] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.093705 14676 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:27.095806 14595 leader_election.cc:341] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.103483 14676 ts_tablet_manager.cc:1428] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.035s user 0.006s sys 0.001s
I20260806 20:34:27.104401 14676 tablet_bootstrap.cc:492] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.110437 14784 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.109777 14675 tablet_bootstrap.cc:492] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:27.113509 14675 ts_tablet_manager.cc:1397] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.015s
I20260806 20:34:27.116468 14784 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.117964 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:27.118621 14632 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.121645 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:34:27.123066 14442 leader_election.cc:341] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.126380 14370 catalog_manager.cc:5495] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:27.128840 14674 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.129294 14674 ts_tablet_manager.cc:1397] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.072s user 0.017s sys 0.000s
I20260806 20:34:27.131654 14674 raft_consensus.cc:348] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.132366 14674 raft_consensus.cc:374] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.132647 14674 raft_consensus.cc:729] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.133249 14674 consensus_queue.cc:260] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.135267 14674 ts_tablet_manager.cc:1428] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 20:34:27.135366 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:27.136085 14674 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.139604 14676 tablet_bootstrap.cc:654] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.141232 14632 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:27.143467 14841 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.143692 14444 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:27.143966 14841 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.145977 14841 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.146188 14675 raft_consensus.cc:348] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.146903 14675 raft_consensus.cc:374] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.147120 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.147284 14675 raft_consensus.cc:729] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.148401 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.148161 14675 consensus_queue.cc:260] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.149220 14554 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:27.150280 14442 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.151113 14675 ts_tablet_manager.cc:1428] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20260806 20:34:27.146991 14674 tablet_bootstrap.cc:654] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.146517 14841 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.152199 14845 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.155484 14675 tablet_bootstrap.cc:492] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:27.156759 14841 raft_consensus.cc:686] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:27.159586 14756 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.160025 14756 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.162037 14756 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.162928 14845 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.163671 14675 tablet_bootstrap.cc:654] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.164606 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:27.166002 14596 leader_election.cc:341] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.157717 14841 consensus_queue.cc:237] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.169838 14784 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.170298 14784 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.172994 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.173738 14554 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:27.142897 14372 catalog_manager.cc:5495] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:27.186556 14784 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.193897 14676 tablet_bootstrap.cc:492] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.194519 14676 ts_tablet_manager.cc:1397] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.090s user 0.034s sys 0.016s
I20260806 20:34:27.197613 14676 raft_consensus.cc:348] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.201515 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.207158 14676 raft_consensus.cc:374] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.204085 14478 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.207472 14676 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.203672 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.208187 14676 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.209206 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.210310 14676 ts_tablet_manager.cc:1428] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260806 20:34:27.212795 14707 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.213194 14676 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.212150 14674 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.213203 14707 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.206833 14878 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:27.219949 14707 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.225554 14877 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 20:34:27.228266 14595 leader_election.cc:341] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.228753 14595 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.229049 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.229831 14478 raft_consensus.cc:2379] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:27.230532 14753 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.206729 14675 tablet_bootstrap.cc:492] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:27.222312 14674 ts_tablet_manager.cc:1397] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.086s user 0.024s sys 0.010s
I20260806 20:34:27.219615 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.231127 14753 consensus_queue.cc:1035] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:27.232062 14370 catalog_manager.cc:5495] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:27.233914 14674 raft_consensus.cc:348] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.234849 14674 raft_consensus.cc:374] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.235114 14674 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.235821 14674 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.236814 14596 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.238449 14756 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.239001 14674 ts_tablet_manager.cc:1428] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260806 20:34:27.239893 14674 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.245352 14674 tablet_bootstrap.cc:654] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.221967 14676 tablet_bootstrap.cc:654] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.258239 14756 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.236837 14675 ts_tablet_manager.cc:1397] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.082s user 0.020s sys 0.009s
I20260806 20:34:27.262328 14675 raft_consensus.cc:348] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.263818 14675 raft_consensus.cc:374] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.264760 14675 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.267565 14675 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.272746 14707 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.273160 14707 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.275929 14877 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:27.277053 14707 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.277309 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:27.278708 14595 leader_election.cc:341] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.279269 14675 ts_tablet_manager.cc:1428] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260806 20:34:27.279719 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.280100 14675 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:27.281103 14827 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.281705 14827 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.285094 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.285959 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.286768 14632 raft_consensus.cc:2379] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:27.288473 14676 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.288900 14676 ts_tablet_manager.cc:1397] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.076s user 0.028s sys 0.018s
I20260806 20:34:27.288861 14827 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.290187 14518 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.291908 14745 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.292268 14675 tablet_bootstrap.cc:654] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:27.292467 14596 leader_election.cc:341] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.292894 14596 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.293988 14755 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.296125 14676 raft_consensus.cc:348] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.296857 14676 raft_consensus.cc:374] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.297176 14676 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.297778 14676 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.298540 14756 consensus_queue.cc:1035] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:27.299835 14676 ts_tablet_manager.cc:1428] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260806 20:34:27.300686 14674 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.301672 14674 ts_tablet_manager.cc:1397] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.062s user 0.019s sys 0.007s
I20260806 20:34:27.300737 14676 tablet_bootstrap.cc:492] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.309353 14674 raft_consensus.cc:348] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.309978 14674 raft_consensus.cc:374] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.310241 14674 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.310423 14745 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.306569 14827 consensus_queue.cc:1035] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:27.310907 14674 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.313223 14676 tablet_bootstrap.cc:654] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.326059 14674 ts_tablet_manager.cc:1428] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260806 20:34:27.326123 14676 tablet_bootstrap.cc:492] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.326915 14676 ts_tablet_manager.cc:1397] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.005s
I20260806 20:34:27.327112 14674 tablet_bootstrap.cc:492] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.330154 14676 raft_consensus.cc:348] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.331099 14676 raft_consensus.cc:374] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.331434 14676 raft_consensus.cc:729] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.333017 14676 consensus_queue.cc:260] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.335891 14676 ts_tablet_manager.cc:1428] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260806 20:34:27.337186 14676 tablet_bootstrap.cc:492] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.346783 14827 consensus_queue.cc:1035] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:27.356688 14676 tablet_bootstrap.cc:654] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.368716 14674 tablet_bootstrap.cc:654] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.378528 14676 tablet_bootstrap.cc:492] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.378991 14676 ts_tablet_manager.cc:1397] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.007s
I20260806 20:34:27.382033 14675 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:27.382591 14675 ts_tablet_manager.cc:1397] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.103s user 0.035s sys 0.038s
I20260806 20:34:27.385411 14675 raft_consensus.cc:348] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.381649 14676 raft_consensus.cc:348] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.386686 14676 raft_consensus.cc:374] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.387106 14676 raft_consensus.cc:729] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.387956 14676 consensus_queue.cc:260] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.388616 14675 raft_consensus.cc:374] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.390030 14675 raft_consensus.cc:729] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.390743 14675 consensus_queue.cc:260] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.391528 14676 ts_tablet_manager.cc:1428] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20260806 20:34:27.399438 14828 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.399931 14828 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.403124 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.392570 14676 tablet_bootstrap.cc:492] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.403613 14478 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.403746 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:27.404842 14521 leader_election.cc:341] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:27.405408 14518 leader_election.cc:341] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.405814 14518 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.406625 14761 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.409070 14675 ts_tablet_manager.cc:1428] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260806 20:34:27.409966 14675 tablet_bootstrap.cc:492] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:27.411069 14676 tablet_bootstrap.cc:654] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.411450 14828 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.414390 14674 tablet_bootstrap.cc:492] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.414880 14674 ts_tablet_manager.cc:1397] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.008s
I20260806 20:34:27.416299 14878 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.416862 14878 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.420236 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.420938 14553 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:27.421407 14878 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.422125 14442 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:27.422995 14878 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.423121 14676 tablet_bootstrap.cc:492] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.423084 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.423494 14878 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.423336 14674 raft_consensus.cc:348] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.423697 14676 ts_tablet_manager.cc:1397] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.003s
I20260806 20:34:27.424141 14674 raft_consensus.cc:374] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.424141 14878 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:27.424376 14444 leader_election.cc:341] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.424649 14674 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.425416 14674 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.426636 14676 raft_consensus.cc:348] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.427281 14676 raft_consensus.cc:374] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.427528 14676 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.427611 14674 ts_tablet_manager.cc:1428] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260806 20:34:27.428664 14674 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.428200 14676 consensus_queue.cc:260] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.430675 14676 ts_tablet_manager.cc:1428] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:34:27.431558 14676 tablet_bootstrap.cc:492] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.433835 14675 tablet_bootstrap.cc:654] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.436829 14676 tablet_bootstrap.cc:654] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.434015 14828 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.437963 14828 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.439988 14828 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.434185 14674 tablet_bootstrap.cc:654] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.451556 14676 tablet_bootstrap.cc:492] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.451953 14676 ts_tablet_manager.cc:1397] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20260806 20:34:27.435396 14878 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.444651 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:27.453615 14518 leader_election.cc:341] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.454182 14878 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.455329 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.455984 14676 raft_consensus.cc:348] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.456698 14676 raft_consensus.cc:374] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.456938 14676 raft_consensus.cc:729] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.457561 14676 consensus_queue.cc:260] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.457995 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:34:27.458478 14521 leader_election.cc:341] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.458763 14553 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.458947 14521 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.459694 14828 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.460417 14676 ts_tablet_manager.cc:1428] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260806 20:34:27.461292 14676 tablet_bootstrap.cc:492] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.464316 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:34:27.465783 14444 leader_election.cc:341] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.468998 14681 consensus_queue.cc:1035] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:27.474586 14676 tablet_bootstrap.cc:654] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.480208 14553 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:27.481443 14442 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.483330 14784 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.483790 14784 raft_consensus.cc:686] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:27.484647 14784 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.485719 14675 tablet_bootstrap.cc:492] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:27.486142 14675 ts_tablet_manager.cc:1397] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.007s
I20260806 20:34:27.488479 14675 raft_consensus.cc:348] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.489131 14675 raft_consensus.cc:374] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.489383 14675 raft_consensus.cc:729] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.489424 14761 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.489940 14761 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.490064 14675 consensus_queue.cc:260] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.490970 14755 consensus_queue.cc:1035] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:27.491979 14761 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.493332 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.493943 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.494827 14675 ts_tablet_manager.cc:1428] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
W20260806 20:34:27.495396 14518 leader_election.cc:341] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.500866 14675 tablet_bootstrap.cc:492] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:27.503481 14755 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.504004 14755 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.506580 14755 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.507454 14675 tablet_bootstrap.cc:654] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.507814 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.507478 14761 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.508934 14761 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.509969 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.511590 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:27.512596 14595 leader_election.cc:341] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:27.512882 14518 leader_election.cc:341] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:27.506736 14521 leader_election.cc:341] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.513739 14761 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.514222 14521 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.516774 14553 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:27.518622 14596 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.526330 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.526713 14755 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.527055 14755 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.527375 14755 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:27.532889 14521 leader_election.cc:341] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.533501 14521 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.534556 14827 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.523766 14761 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.539803 14828 consensus_queue.cc:1035] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:27.548774 14755 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.554112 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:34:27.556936 14595 leader_election.cc:341] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.560807 14681 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:27.567428 14372 catalog_manager.cc:5495] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:27.571770 14676 tablet_bootstrap.cc:492] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.572250 14676 ts_tablet_manager.cc:1397] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.111s user 0.041s sys 0.030s
I20260806 20:34:27.573086 14755 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.574776 14676 raft_consensus.cc:348] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.575556 14676 raft_consensus.cc:374] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.575826 14676 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.576478 14676 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.579069 14674 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.579602 14674 ts_tablet_manager.cc:1397] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.151s user 0.011s sys 0.023s
I20260806 20:34:27.579490 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:27.582505 14674 raft_consensus.cc:348] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.582903 14553 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.583281 14674 raft_consensus.cc:374] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.583562 14674 raft_consensus.cc:729] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.584125 14674 consensus_queue.cc:260] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.586407 14674 ts_tablet_manager.cc:1428] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 20:34:27.587265 14674 tablet_bootstrap.cc:492] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.592195 14676 ts_tablet_manager.cc:1428] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260806 20:34:27.594429 14674 tablet_bootstrap.cc:654] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.594879 14553 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:27.595949 14596 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.597097 14755 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.597553 14755 raft_consensus.cc:686] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:27.598325 14755 consensus_queue.cc:237] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.601857 14676 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.608180 14676 tablet_bootstrap.cc:654] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.612344 14707 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 20:34:27.617975 14518 consensus_peers.cc:487] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Couldn't send request to peer 92828dd6dbe74f29a04d4e702b502621. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:27.623620 14675 tablet_bootstrap.cc:492] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:27.624042 14674 tablet_bootstrap.cc:492] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.625181 14674 ts_tablet_manager.cc:1397] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.016s
I20260806 20:34:27.624115 14675 ts_tablet_manager.cc:1397] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.123s user 0.010s sys 0.001s
I20260806 20:34:27.627558 14674 raft_consensus.cc:348] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.628252 14674 raft_consensus.cc:374] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.628508 14674 raft_consensus.cc:729] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.629115 14674 consensus_queue.cc:260] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.631150 14828 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.631624 14828 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.636089 14675 raft_consensus.cc:348] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.636811 14675 raft_consensus.cc:374] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.637094 14675 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.636893 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.637792 14632 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:27.640067 14828 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.640228 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.638605 14675 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.639732 14761 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.641996 14761 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.643237 14674 ts_tablet_manager.cc:1428] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.018s user 0.003s sys 0.002s
I20260806 20:34:27.643950 14761 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.641028 14521 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:34:27.645344 14518 leader_election.cc:341] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.645022 14372 catalog_manager.cc:5495] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:27.648577 14761 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.648957 14761 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.649302 14761 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.650408 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:27.654824 14518 leader_election.cc:341] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.644079 14674 tablet_bootstrap.cc:492] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.645040 14676 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.655768 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.656318 14676 ts_tablet_manager.cc:1397] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.001s
I20260806 20:34:27.656426 14632 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:27.655781 14761 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.658648 14676 raft_consensus.cc:348] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.659454 14676 raft_consensus.cc:374] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.659716 14676 raft_consensus.cc:729] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.660360 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:27.660853 14676 consensus_queue.cc:260] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:27.661645 14518 leader_election.cc:341] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.661890 14675 ts_tablet_manager.cc:1428] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260806 20:34:27.662495 14761 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.662760 14675 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:27.662519 14676 ts_tablet_manager.cc:1428] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 20:34:27.658593 14521 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.664067 14761 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.664394 14761 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.664733 14761 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.665526 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:27.665810 14676 tablet_bootstrap.cc:492] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.666093 14632 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.672139 14674 tablet_bootstrap.cc:654] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.679296 14632 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:27.680608 14521 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:27.681489 14828 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.681972 14828 raft_consensus.cc:686] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:27.683440 14674 tablet_bootstrap.cc:492] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.684056 14674 ts_tablet_manager.cc:1397] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.001s
I20260806 20:34:27.684273 14755 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.684362 14675 tablet_bootstrap.cc:654] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.683127 14828 consensus_queue.cc:237] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.685005 14755 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.686957 14755 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.687911 14745 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.688107 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.688829 14676 tablet_bootstrap.cc:654] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.688411 14745 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.689154 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.689771 14478 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:27.689942 14674 raft_consensus.cc:348] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.690372 14745 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.690698 14674 raft_consensus.cc:374] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.691026 14674 raft_consensus.cc:729] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.691586 14674 consensus_queue.cc:260] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.692225 14595 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:27.693730 14674 ts_tablet_manager.cc:1428] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 20:34:27.695796 14674 tablet_bootstrap.cc:492] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.697335 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.688829 14554 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:27.699889 14707 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.700235 14707 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.700601 14707 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.701387 14674 tablet_bootstrap.cc:654] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.704511 14372 catalog_manager.cc:5495] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:27.709594 14707 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.712226 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:27.712831 14477 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.705402 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.717968 14784 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.718420 14784 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.720314 14784 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.721227 14841 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:27.723234 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.724120 14553 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:27.725924 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.726722 14632 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:27.728155 14477 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:27.728171 14444 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:27.729290 14595 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:27.729692 14784 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.730139 14755 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.730672 14755 raft_consensus.cc:686] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:27.731490 14755 consensus_queue.cc:237] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.730166 14784 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.732982 14784 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.736644 14676 tablet_bootstrap.cc:492] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.737094 14676 ts_tablet_manager.cc:1397] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.001s
W20260806 20:34:27.706360 14521 leader_election.cc:341] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:27.738930 14518 leader_election.cc:341] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.739744 14518 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.739619 14676 raft_consensus.cc:348] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.740304 14676 raft_consensus.cc:374] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.740590 14828 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.740694 14676 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.741302 14676 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.745903 14676 ts_tablet_manager.cc:1428] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 20:34:27.746763 14676 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.750806 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:27.751578 14554 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.752380 14676 tablet_bootstrap.cc:654] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.754822 14707 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:27.750586 14761 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.756633 14784 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.761904 14674 tablet_bootstrap.cc:492] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.762331 14674 ts_tablet_manager.cc:1397] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.019s
I20260806 20:34:27.764496 14372 catalog_manager.cc:5495] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:27.758854 14745 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.769135 14745 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.771063 14745 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.772184 14877 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:27.773082 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:27.772992 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.773627 14628 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.771329 14761 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:27.775256 14521 leader_election.cc:341] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.779400 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.780561 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:27.780864 14628 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:27.781881 14521 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:34:27.783053 14518 leader_election.cc:341] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.783478 14518 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.784262 14828 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:34:27.784628 14518 leader_election.cc:341] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.784754 14828 raft_consensus.cc:686] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:27.787186 14745 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.789775 14828 consensus_queue.cc:237] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.827158 14554 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:27.831894 14674 raft_consensus.cc:348] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.832664 14674 raft_consensus.cc:374] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.832928 14674 raft_consensus.cc:729] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.833482 14674 consensus_queue.cc:260] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.835567 14675 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:27.836052 14675 ts_tablet_manager.cc:1397] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.173s user 0.033s sys 0.020s
I20260806 20:34:27.837704 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:27.838284 14553 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.839135 14674 ts_tablet_manager.cc:1428] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.077s user 0.006s sys 0.002s
I20260806 20:34:27.840008 14674 tablet_bootstrap.cc:492] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.843159 14675 raft_consensus.cc:348] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.844286 14675 raft_consensus.cc:374] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.844609 14675 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.842738 14745 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.846654 14878 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.846493 14745 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.847117 14878 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.848621 14745 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.850415 14553 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:27.855105 14442 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:27.855345 14878 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.856051 14878 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.854282 14675 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.853504 14826 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.861531 14877 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:27.861934 14826 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.862103 14877 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.864125 14878 raft_consensus.cc:686] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:27.864809 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.865087 14878 consensus_queue.cc:237] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.865401 14826 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.865497 14877 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.865564 14555 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:27.867628 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.863619 14674 tablet_bootstrap.cc:654] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.874119 14675 ts_tablet_manager.cc:1428] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.038s user 0.004s sys 0.004s
I20260806 20:34:27.875367 14784 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.869068 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.877312 14442 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:27.878583 14877 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.878513 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:27.881731 14675 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:27.879052 14877 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.887751 14877 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.893242 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.894170 14479 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:27.895722 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:27.896574 14555 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:27.903643 14877 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.904829 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:27.906140 14518 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:27.906908 14826 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.907244 14826 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:34:27.907325 14518 leader_election.cc:341] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.907629 14826 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.911115 14372 catalog_manager.cc:5495] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:27.909708 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:27.914698 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:27.915284 14552 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.914095 14841 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:27.918093 14841 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:27.918788 14841 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.909423 14877 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:27.908887 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 43718us
I20260806 20:34:27.924430 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:27.918385 14521 leader_election.cc:341] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:27.931221 14521 leader_election.cc:341] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.931653 14521 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:27.934566 14676 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:27.935902 14676 ts_tablet_manager.cc:1397] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.189s user 0.026s sys 0.001s
I20260806 20:34:27.935333 14828 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:27.943398 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:27.943992 14632 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.950344 14632 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:27.951740 14676 raft_consensus.cc:348] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.952173 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:27.952733 14676 raft_consensus.cc:374] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.953095 14676 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.954154 14676 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.939380 14841 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:27.952206 14444 leader_election.cc:341] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.956911 14676 ts_tablet_manager.cc:1428] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260806 20:34:27.957873 14676 tablet_bootstrap.cc:492] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:27.960523 14552 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:27.955387 14826 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.962718 14442 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:27.963703 14878 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:27.964336 14878 raft_consensus.cc:686] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:27.965536 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:27.966354 14479 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:27.966787 14878 consensus_queue.cc:237] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.967928 14674 tablet_bootstrap.cc:492] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:27.968400 14674 ts_tablet_manager.cc:1397] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.129s user 0.030s sys 0.011s
I20260806 20:34:27.971576 14674 raft_consensus.cc:348] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.972298 14674 raft_consensus.cc:374] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:27.972589 14674 raft_consensus.cc:729] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:27.973160 14674 consensus_queue.cc:260] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:27.973539 14826 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.973698 14675 tablet_bootstrap.cc:654] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:27.970176 14444 leader_election.cc:341] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.974864 14676 tablet_bootstrap.cc:654] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.975765 14674 ts_tablet_manager.cc:1428] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:34:27.975696 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:27.977622 14674 tablet_bootstrap.cc:492] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:27.981006 14479 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:27.982610 14841 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:27.984205 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:27.985363 14552 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:27.991607 14674 tablet_bootstrap.cc:654] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:27.993574 14681 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 20:34:27.998153 14521 leader_election.cc:341] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:27.998708 14521 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:34:27.999718 14444 leader_election.cc:341] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.002345 14630 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.005293 14675 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.005757 14675 ts_tablet_manager.cc:1397] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.124s user 0.026s sys 0.008s
I20260806 20:34:28.008672 14675 raft_consensus.cc:348] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.009229 14826 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.009307 14675 raft_consensus.cc:374] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.009682 14675 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.010886 14675 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.013427 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:28.015669 14675 ts_tablet_manager.cc:1428] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260806 20:34:28.017223 14675 tablet_bootstrap.cc:492] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
W20260806 20:34:28.017370 14444 leader_election.cc:341] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.019320 14444 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.024631 14925 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.019757 14877 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.027024 14674 tablet_bootstrap.cc:492] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.027477 14674 ts_tablet_manager.cc:1397] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.010s
I20260806 20:34:28.029757 14675 tablet_bootstrap.cc:654] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.029901 14674 raft_consensus.cc:348] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.030709 14674 raft_consensus.cc:374] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.030975 14674 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.031766 14674 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.032284 14707 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.034689 14674 ts_tablet_manager.cc:1428] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:34:28.035539 14674 tablet_bootstrap.cc:492] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.040943 14877 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:28.021193 14676 tablet_bootstrap.cc:492] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:28.042126 14676 ts_tablet_manager.cc:1397] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.000s
I20260806 20:34:28.044631 14676 raft_consensus.cc:348] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.045395 14676 raft_consensus.cc:374] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.045652 14676 raft_consensus.cc:729] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.046219 14676 consensus_queue.cc:260] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.047160 14674 tablet_bootstrap.cc:654] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.049345 14676 ts_tablet_manager.cc:1428] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:34:28.050263 14676 tablet_bootstrap.cc:492] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:28.057284 14676 tablet_bootstrap.cc:654] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.061480 14479 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.062003 14675 tablet_bootstrap.cc:492] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.062486 14675 ts_tablet_manager.cc:1397] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.004s
I20260806 20:34:28.065138 14675 raft_consensus.cc:348] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.065838 14675 raft_consensus.cc:374] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.066109 14675 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.066720 14675 consensus_queue.cc:260] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.071159 14675 ts_tablet_manager.cc:1428] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 20:34:28.072099 14675 tablet_bootstrap.cc:492] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.072513 14372 catalog_manager.cc:5495] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:28.074374 14372 catalog_manager.cc:5495] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:28.081579 14826 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.082127 14826 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.083338 14675 tablet_bootstrap.cc:654] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.085328 14707 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.085727 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.085785 14707 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.087881 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.084209 14826 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:28.088574 14518 leader_election.cc:341] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.091496 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.092582 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:28.093190 14595 leader_election.cc:341] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.093400 14554 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:28.095072 14596 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.088287 14707 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.088577 14628 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:28.098327 14756 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.098961 14521 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.101624 14826 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.102038 14826 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.102386 14826 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.101619 14674 tablet_bootstrap.cc:492] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.105080 14674 ts_tablet_manager.cc:1397] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.070s user 0.015s sys 0.005s
I20260806 20:34:28.107468 14674 raft_consensus.cc:348] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.108155 14674 raft_consensus.cc:374] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.108438 14674 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.108865 14826 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.109169 14674 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.109596 14676 tablet_bootstrap.cc:492] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:28.111227 14674 ts_tablet_manager.cc:1428] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:34:28.111663 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:28.112114 14674 tablet_bootstrap.cc:492] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
W20260806 20:34:28.112754 14518 leader_election.cc:341] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.114929 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:28.115520 14630 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.119469 14826 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.122207 14630 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:28.119640 14674 tablet_bootstrap.cc:654] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.122782 14756 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.123334 14756 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:28.123610 14521 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.125108 14841 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.125628 14841 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.127665 14841 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.129956 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.130806 14554 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:28.133951 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.134264 14826 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:28.134948 14674 tablet_bootstrap.cc:492] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.135290 14442 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:28.135406 14674 ts_tablet_manager.cc:1397] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.003s
I20260806 20:34:28.134944 14826 raft_consensus.cc:686] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:34:28.136268 14444 leader_election.cc:341] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.136420 14826 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.137429 14841 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.139779 14841 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.140300 14841 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.143646 14676 ts_tablet_manager.cc:1397] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.094s user 0.016s sys 0.003s
I20260806 20:34:28.159154 14828 consensus_queue.cc:1035] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 20:34:28.163645 14841 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.168915 14681 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.166375 14676 raft_consensus.cc:348] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.175791 14676 raft_consensus.cc:374] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.176777 14372 catalog_manager.cc:5495] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:28.138837 14756 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.138023 14674 raft_consensus.cc:348] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.176095 14676 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
W20260806 20:34:28.183427 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:28.197753 14674 raft_consensus.cc:374] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.198233 14841 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.184218 14675 tablet_bootstrap.cc:492] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.198042 14676 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.190526 14745 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.199378 14745 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.198999 14675 ts_tablet_manager.cc:1397] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.127s user 0.045s sys 0.021s
I20260806 20:34:28.200397 14676 ts_tablet_manager.cc:1428] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.056s user 0.004s sys 0.002s
I20260806 20:34:28.201344 14676 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:28.201536 14745 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.202502 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:28.202678 14675 raft_consensus.cc:348] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.203418 14675 raft_consensus.cc:374] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.203678 14675 raft_consensus.cc:729] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
W20260806 20:34:28.204130 14444 leader_election.cc:341] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.204319 14675 consensus_queue.cc:260] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:28.193842 14518 consensus_peers.cc:487] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Couldn't send request to peer 92828dd6dbe74f29a04d4e702b502621. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:28.205458 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.206410 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:34:28.206796 14521 leader_election.cc:341] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.207026 14675 ts_tablet_manager.cc:1428] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:34:28.207913 14675 tablet_bootstrap.cc:492] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.207000 14552 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.217653 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:28.229472 14675 tablet_bootstrap.cc:654] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.220882 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:28.208134 14674 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
W20260806 20:34:28.231231 14595 leader_election.cc:341] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.224570 14552 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:28.231421 14674 consensus_queue.cc:260] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.231681 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.233170 14681 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:34:28.237406 14442 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.240798 14841 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:28.241285 14841 raft_consensus.cc:686] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:28.242075 14841 consensus_queue.cc:237] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.245281 14878 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.246191 14878 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.250430 14675 tablet_bootstrap.cc:492] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.250932 14675 ts_tablet_manager.cc:1397] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.016s
I20260806 20:34:28.251215 14878 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.252722 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.253540 14628 raft_consensus.cc:2379] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:34:28.253371 14675 raft_consensus.cc:348] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.254050 14675 raft_consensus.cc:374] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.254329 14675 raft_consensus.cc:729] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.256786 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.256897 14675 consensus_queue.cc:260] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.209169 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:28.264284 14518 leader_election.cc:341] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.264766 14518 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.265836 14745 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.258208 14442 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.225945 14756 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.266708 14681 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.267315 14878 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.258754 14675 ts_tablet_manager.cc:1428] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 20:34:28.268606 14675 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.267458 14674 ts_tablet_manager.cc:1428] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.132s user 0.005s sys 0.001s
I20260806 20:34:28.261263 14676 tablet_bootstrap.cc:654] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.274601 14674 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.276824 14675 tablet_bootstrap.cc:654] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.279734 14878 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.284689 14370 catalog_manager.cc:5495] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:28.288661 14745 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.290321 14674 tablet_bootstrap.cc:654] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.308995 14675 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.309465 14675 ts_tablet_manager.cc:1397] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.013s
I20260806 20:34:28.312645 14675 raft_consensus.cc:348] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.313449 14675 raft_consensus.cc:374] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.313747 14675 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.314471 14675 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.316955 14675 ts_tablet_manager.cc:1428] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 20:34:28.317745 14674 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.317826 14675 tablet_bootstrap.cc:492] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.318387 14674 ts_tablet_manager.cc:1397] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.001s
I20260806 20:34:28.319139 14676 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:28.319568 14676 ts_tablet_manager.cc:1397] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.118s user 0.006s sys 0.008s
I20260806 20:34:28.321040 14674 raft_consensus.cc:348] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.322008 14674 raft_consensus.cc:374] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.322399 14674 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.323295 14674 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.326882 14674 ts_tablet_manager.cc:1428] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260806 20:34:28.325806 14676 raft_consensus.cc:348] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.327355 14675 tablet_bootstrap.cc:654] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.332043 14676 raft_consensus.cc:374] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.332384 14676 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.333123 14676 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.329031 14674 tablet_bootstrap.cc:492] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.335314 14676 ts_tablet_manager.cc:1428] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.015s user 0.009s sys 0.001s
I20260806 20:34:28.336134 14676 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:28.338773 14745 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:28.340045 14674 tablet_bootstrap.cc:654] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.340580 14756 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.341123 14756 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.341836 14676 tablet_bootstrap.cc:654] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.343330 14756 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.347597 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.348083 14675 tablet_bootstrap.cc:492] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.348429 14554 raft_consensus.cc:2379] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:28.348542 14675 ts_tablet_manager.cc:1397] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.013s
I20260806 20:34:28.353345 14878 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.353883 14878 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.354631 14674 tablet_bootstrap.cc:492] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.355020 14674 ts_tablet_manager.cc:1397] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.006s
I20260806 20:34:28.355346 14676 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:28.355818 14676 ts_tablet_manager.cc:1397] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.000s
I20260806 20:34:28.356732 14878 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.356984 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.357172 14674 raft_consensus.cc:348] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.357636 14633 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:28.357810 14674 raft_consensus.cc:374] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.358063 14674 raft_consensus.cc:729] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.358151 14676 raft_consensus.cc:348] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.358956 14676 raft_consensus.cc:374] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.358831 14444 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:28.358681 14674 consensus_queue.cc:260] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.359270 14676 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.359756 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.360244 14925 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.360018 14676 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.361052 14925 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.361493 14925 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.362711 14675 raft_consensus.cc:348] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.363140 14755 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.363415 14675 raft_consensus.cc:374] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.363746 14675 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.363631 14755 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.364318 14595 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.365770 14755 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.364420 14675 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.366488 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.367103 14554 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:28.365470 14756 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.367703 14675 ts_tablet_manager.cc:1428] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.019s user 0.002s sys 0.004s
I20260806 20:34:28.368470 14925 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.369405 14675 tablet_bootstrap.cc:492] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.374130 14674 ts_tablet_manager.cc:1428] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260806 20:34:28.374394 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:28.375003 14925 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.375053 14630 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.376420 14675 tablet_bootstrap.cc:654] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.375717 14674 tablet_bootstrap.cc:492] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.363102 14676 ts_tablet_manager.cc:1428] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260806 20:34:28.381259 14676 tablet_bootstrap.cc:492] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:28.393873 14841 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:28.396229 14826 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.396845 14826 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.398929 14826 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.393898 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:28.401330 14554 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.402412 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.405725 14674 tablet_bootstrap.cc:654] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.408929 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.409735 14555 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:34:28.410295 14521 leader_election.cc:341] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.411453 14841 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:28.412609 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.413584 14878 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:34:28.415655 14518 leader_election.cc:341] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.416147 14518 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.418962 14826 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.423517 14554 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:28.391716 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.425120 14479 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:28.426362 14595 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.427598 14755 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.427768 14676 tablet_bootstrap.cc:654] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.439697 14442 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:28.441375 14841 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:28.441929 14841 raft_consensus.cc:686] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:28.442936 14841 consensus_queue.cc:237] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.447603 14675 tablet_bootstrap.cc:492] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.448249 14675 ts_tablet_manager.cc:1397] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.080s user 0.017s sys 0.031s
I20260806 20:34:28.451592 14755 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.453819 14756 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.456249 14630 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:28.460378 14674 tablet_bootstrap.cc:492] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.460908 14674 ts_tablet_manager.cc:1397] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.086s user 0.016s sys 0.005s
I20260806 20:34:28.475581 14674 raft_consensus.cc:348] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.476419 14674 raft_consensus.cc:374] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.476802 14674 raft_consensus.cc:729] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.477679 14674 consensus_queue.cc:260] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.480201 14674 ts_tablet_manager.cc:1428] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.019s user 0.004s sys 0.003s
I20260806 20:34:28.481345 14674 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.481633 14924 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.482298 14924 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.487599 14674 tablet_bootstrap.cc:654] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.491647 14877 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:28.497702 14924 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.498435 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.499171 14555 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:28.499342 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.500385 14442 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:28.451495 14675 raft_consensus.cc:348] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.502348 14675 raft_consensus.cc:374] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.502717 14674 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.501255 14924 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.503180 14674 ts_tablet_manager.cc:1397] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.007s
I20260806 20:34:28.503249 14924 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.504040 14924 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:28.500972 14444 leader_election.cc:341] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.506083 14674 raft_consensus.cc:348] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.506788 14674 raft_consensus.cc:374] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.507071 14674 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.507668 14674 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.512765 14674 ts_tablet_manager.cc:1428] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260806 20:34:28.515168 14745 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.516029 14674 tablet_bootstrap.cc:492] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.515786 14745 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.519511 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:28.521341 14518 leader_election.cc:341] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.523643 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:28.525333 14521 leader_election.cc:341] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.525768 14521 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.526752 14827 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.527139 14674 tablet_bootstrap.cc:654] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.534049 14745 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.534735 14841 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.536600 14841 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.538691 14841 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.539561 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.539678 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.542372 14676 tablet_bootstrap.cc:492] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:28.542826 14676 ts_tablet_manager.cc:1397] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.162s user 0.012s sys 0.039s
I20260806 20:34:28.546193 14676 raft_consensus.cc:348] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.546854 14676 raft_consensus.cc:374] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.547132 14676 raft_consensus.cc:729] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.502662 14675 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.548851 14442 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.549937 14841 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.547724 14676 consensus_queue.cc:260] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.551478 14676 ts_tablet_manager.cc:1428] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260806 20:34:28.535571 14674 tablet_bootstrap.cc:492] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.551559 14924 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.552379 14675 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.560393 14675 ts_tablet_manager.cc:1428] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.112s user 0.006s sys 0.000s
I20260806 20:34:28.561311 14675 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.570207 14675 tablet_bootstrap.cc:654] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.552090 14674 ts_tablet_manager.cc:1397] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.006s
I20260806 20:34:28.553589 14924 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.554551 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:28.554602 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:28.583184 14553 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.585716 14674 raft_consensus.cc:348] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.586449 14674 raft_consensus.cc:374] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.586699 14674 raft_consensus.cc:729] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.587265 14674 consensus_queue.cc:260] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:28.588340 14444 leader_election.cc:341] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.553014 14676 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:28.589370 14674 ts_tablet_manager.cc:1428] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:34:28.590278 14674 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.595678 14674 tablet_bootstrap.cc:654] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.596532 14676 tablet_bootstrap.cc:654] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.601930 14553 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:28.604166 14442 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.605018 14924 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:28.605479 14924 raft_consensus.cc:686] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:28.606312 14924 consensus_queue.cc:237] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.606961 14675 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.607525 14675 ts_tablet_manager.cc:1397] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.000s
I20260806 20:34:28.613591 14675 raft_consensus.cc:348] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.611199 14676 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:28.625192 14676 ts_tablet_manager.cc:1397] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.002s
I20260806 20:34:28.629354 14676 raft_consensus.cc:348] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.634768 14676 raft_consensus.cc:374] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.635210 14675 raft_consensus.cc:374] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.635066 14369 catalog_manager.cc:5495] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:28.635135 14676 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.637552 14675 raft_consensus.cc:729] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.637356 14676 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.641364 14675 consensus_queue.cc:260] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.644253 14479 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.646113 14674 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.647212 14674 ts_tablet_manager.cc:1397] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.057s user 0.021s sys 0.032s
I20260806 20:34:28.650067 14674 raft_consensus.cc:348] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.650563 14674 raft_consensus.cc:374] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.650794 14674 raft_consensus.cc:729] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.651383 14674 consensus_queue.cc:260] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.653323 14674 ts_tablet_manager.cc:1428] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 20:34:28.663640 14674 tablet_bootstrap.cc:492] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.663825 14675 ts_tablet_manager.cc:1428] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20260806 20:34:28.664599 14826 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.669134 14674 tablet_bootstrap.cc:654] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.674165 14676 ts_tablet_manager.cc:1428] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.047s user 0.005s sys 0.001s
I20260806 20:34:28.664891 14675 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.675334 14676 tablet_bootstrap.cc:492] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:28.682125 14674 tablet_bootstrap.cc:492] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.682868 14675 tablet_bootstrap.cc:654] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.682699 14674 ts_tablet_manager.cc:1397] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260806 20:34:28.687747 14674 raft_consensus.cc:348] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.688477 14674 raft_consensus.cc:374] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.688717 14674 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.689298 14674 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.695727 14674 ts_tablet_manager.cc:1428] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260806 20:34:28.696305 14676 tablet_bootstrap.cc:654] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.696636 14674 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.701924 14924 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.706756 14924 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.710275 14755 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.710763 14755 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.702905 14628 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.713901 14755 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.714103 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.702209 14370 catalog_manager.cc:5495] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:28.716686 14630 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
W20260806 20:34:28.718315 14442 leader_election.cc:341] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.718489 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.719103 14479 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:28.721020 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:28.721673 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.722087 14755 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.722432 14755 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.722857 14755 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.722918 14675 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.723579 14675 ts_tablet_manager.cc:1397] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.003s
I20260806 20:34:28.724762 14826 consensus_queue.cc:1046] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.725265 14674 tablet_bootstrap.cc:654] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.722383 14553 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:28.728871 14745 raft_consensus.cc:3032] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:28.729406 14745 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:28.730172 14745 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.728822 14877 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:28.732570 14755 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.734162 14675 raft_consensus.cc:348] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.734956 14745 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:28.734966 14675 raft_consensus.cc:374] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.735523 14675 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.735425 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:28.736755 14826 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.736981 14477 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.737210 14826 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.736326 14675 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.743428 14826 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.744027 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:28.745398 14521 leader_election.cc:341] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.747049 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:28.747450 14924 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.747622 14553 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.749068 14675 ts_tablet_manager.cc:1428] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260806 20:34:28.750154 14675 tablet_bootstrap.cc:492] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.754920 14553 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:28.748974 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.753134 14826 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.755738 14826 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:28.756410 14444 leader_election.cc:341] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.756858 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.758881 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.759545 14477 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:28.759801 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:28.760239 14518 leader_election.cc:341] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.760325 14755 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.760247 14826 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.761482 14596 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:28.764537 14676 tablet_bootstrap.cc:492] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
W20260806 20:34:28.775157 14521 leader_election.cc:341] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.764643 14924 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.778494 14675 tablet_bootstrap.cc:654] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.773510 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.775431 14676 ts_tablet_manager.cc:1397] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.009s
I20260806 20:34:28.775605 14521 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.782157 14683 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.777976 14755 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:34:28.791285 14518 leader_election.cc:341] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.791795 14518 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.793641 14676 raft_consensus.cc:348] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.794173 14745 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.794642 14676 raft_consensus.cc:374] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.795434 14676 raft_consensus.cc:729] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.796120 14676 consensus_queue.cc:260] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.799095 14755 raft_consensus.cc:686] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:28.804303 14676 ts_tablet_manager.cc:1428] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260806 20:34:28.804647 14707 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.805274 14676 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:28.805075 14707 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.806743 14707 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.804358 14755 consensus_queue.cc:237] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.813261 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.816174 14674 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.816665 14674 ts_tablet_manager.cc:1397] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.120s user 0.011s sys 0.031s
I20260806 20:34:28.823647 14674 raft_consensus.cc:348] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.824270 14674 raft_consensus.cc:374] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.825177 14674 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.825826 14674 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.831519 14674 ts_tablet_manager.cc:1428] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
W20260806 20:34:28.836148 14595 leader_election.cc:341] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.824695 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.837904 14553 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:28.854254 14596 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.842195 14878 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260806 20:34:28.861394 14707 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.861842 14707 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.864609 14707 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.848726 14477 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.878738 14674 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:28.875939 14877 consensus_queue.cc:1035] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 20:34:28.895960 14707 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.897321 14707 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.904878 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:28.905644 14555 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.902607 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:28.910231 14675 tablet_bootstrap.cc:492] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:28.913343 14674 tablet_bootstrap.cc:654] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.913770 14675 ts_tablet_manager.cc:1397] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.164s user 0.029s sys 0.028s
W20260806 20:34:28.914368 14595 leader_election.cc:341] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.911322 14676 tablet_bootstrap.cc:654] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.912312 14683 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.915238 14683 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.913050 14555 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:28.917364 14683 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.908234 14372 catalog_manager.cc:5495] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:28.916697 14681 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.922773 14596 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:28.923568 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.923908 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:28.925349 14521 leader_election.cc:341] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.925606 14755 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:28.926251 14755 raft_consensus.cc:686] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:28.922340 14681 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.927155 14755 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:28.926282 14518 leader_election.cc:341] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:28.928901 14518 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:28.931437 14683 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:28.937122 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.940824 14675 raft_consensus.cc:348] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.942068 14675 raft_consensus.cc:374] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.942323 14675 raft_consensus.cc:729] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.928355 14681 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:28.937995 14554 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:28.950871 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:28.951565 14478 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:28.954070 14924 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:28.955534 14595 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:28.954111 14675 consensus_queue.cc:260] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.955534 14924 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.959550 14924 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:28.958078 14681 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.964251 14681 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.964645 14681 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.964994 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:28.965616 14628 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:28.972992 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:28.973621 14555 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:28.978991 14674 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:28.979422 14674 ts_tablet_manager.cc:1397] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.101s user 0.011s sys 0.004s
I20260806 20:34:28.981387 14675 ts_tablet_manager.cc:1428] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20260806 20:34:28.982317 14675 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:28.983637 14676 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:28.983263 14674 raft_consensus.cc:348] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.986680 14674 raft_consensus.cc:374] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.987018 14674 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.987697 14674 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.988374 14877 consensus_queue.cc:1035] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:28.990057 14674 ts_tablet_manager.cc:1428] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260806 20:34:28.983809 14442 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:28.991434 14676 ts_tablet_manager.cc:1397] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.186s user 0.010s sys 0.013s
I20260806 20:34:28.991839 14924 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:28.992272 14924 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:28.992635 14924 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:28.993856 14676 raft_consensus.cc:348] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.993813 14681 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.994446 14676 raft_consensus.cc:374] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:28.994736 14676 raft_consensus.cc:729] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:28.995362 14676 consensus_queue.cc:260] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:28.997507 14676 ts_tablet_manager.cc:1428] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:34:28.998335 14675 tablet_bootstrap.cc:654] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:28.999635 14674 tablet_bootstrap.cc:492] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:29.000029 14676 tablet_bootstrap.cc:492] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:29.005851 14674 tablet_bootstrap.cc:654] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.009654 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:29.010293 14477 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.011788 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:29.012341 14554 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.013128 14681 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:29.018265 14554 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:29.021683 14477 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:29.021597 14596 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:29.022614 14681 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:29.023300 14681 raft_consensus.cc:686] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:29.024013 14681 consensus_queue.cc:237] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.018873 14924 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.029381 14676 tablet_bootstrap.cc:654] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.029762 14924 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.036365 14369 catalog_manager.cc:5495] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
W20260806 20:34:29.043143 14444 consensus_peers.cc:487] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 -> Peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Couldn't send request to peer df1245ecd2034e11ae6cf688c1c10a4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:29.044214 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:29.044984 14554 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.041587 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:29.053006 14628 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.056113 14756 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.058697 14756 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.060288 14675 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:29.060853 14554 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:29.062729 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:29.063395 14477 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:29.063679 14442 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:29.064522 14924 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:29.064574 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:29.065091 14924 raft_consensus.cc:686] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:29.065448 14755 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:29.065784 14755 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:29.066152 14755 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.065932 14924 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.060906 14675 ts_tablet_manager.cc:1397] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.079s user 0.015s sys 0.009s
I20260806 20:34:29.061277 14756 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:29.073096 14675 raft_consensus.cc:348] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.073767 14675 raft_consensus.cc:374] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.074126 14675 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.074870 14675 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.078006 14675 ts_tablet_manager.cc:1428] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260806 20:34:29.079071 14675 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:29.093374 14674 tablet_bootstrap.cc:492] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:29.093858 14674 ts_tablet_manager.cc:1397] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.094s user 0.012s sys 0.014s
I20260806 20:34:29.096029 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:29.099059 14596 leader_election.cc:341] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.101181 14676 tablet_bootstrap.cc:492] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:29.101647 14676 ts_tablet_manager.cc:1397] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.102s user 0.012s sys 0.007s
I20260806 20:34:29.105141 14675 tablet_bootstrap.cc:654] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.109145 14674 raft_consensus.cc:348] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.109905 14674 raft_consensus.cc:374] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.110252 14674 raft_consensus.cc:729] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.111136 14674 consensus_queue.cc:260] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.121407 14755 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.123757 14674 ts_tablet_manager.cc:1428] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260806 20:34:29.124756 14674 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:29.127249 14755 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:29.128580 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:29.129233 14478 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.129689 14676 raft_consensus.cc:348] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.130389 14676 raft_consensus.cc:374] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.130504 14674 tablet_bootstrap.cc:654] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.130657 14676 raft_consensus.cc:729] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.131418 14676 consensus_queue.cc:260] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.132805 14369 catalog_manager.cc:5495] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:29.133581 14676 ts_tablet_manager.cc:1428] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260806 20:34:29.134471 14676 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:29.139945 14676 tablet_bootstrap.cc:654] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.152084 14676 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:29.159056 14676 ts_tablet_manager.cc:1397] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.003s
I20260806 20:34:29.162230 14676 raft_consensus.cc:348] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.167034 14676 raft_consensus.cc:374] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.167308 14676 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.167873 14676 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.169494 14676 ts_tablet_manager.cc:1428] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 20:34:29.174371 14676 tablet_bootstrap.cc:492] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:29.179981 14676 tablet_bootstrap.cc:654] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.186887 14924 consensus_queue.cc:1035] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:29.191030 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:29.191695 14555 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.192682 14674 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:29.193148 14674 ts_tablet_manager.cc:1397] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.069s user 0.028s sys 0.036s
I20260806 20:34:29.195693 14674 raft_consensus.cc:348] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.196338 14674 raft_consensus.cc:374] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.196616 14674 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.196815 14924 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.197228 14674 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.197417 14924 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.200934 14924 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.201939 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:29.202713 14632 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:29.204121 14444 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:29.205293 14674 ts_tablet_manager.cc:1428] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 20:34:29.206189 14674 tablet_bootstrap.cc:492] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:29.208245 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:29.208961 14553 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:29.211540 14674 tablet_bootstrap.cc:654] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.212415 14745 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.212949 14745 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.218173 14745 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.218417 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:29.219992 14521 leader_election.cc:341] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.228864 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:29.230227 14674 tablet_bootstrap.cc:492] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
W20260806 20:34:29.230497 14518 leader_election.cc:341] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.230993 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:29.120106 14675 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:29.121471 14628 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:29.232764 14675 ts_tablet_manager.cc:1397] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.154s user 0.013s sys 0.006s
I20260806 20:34:29.230671 14674 ts_tablet_manager.cc:1397] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.005s
I20260806 20:34:29.235219 14675 raft_consensus.cc:348] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.236035 14745 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:29.235899 14675 raft_consensus.cc:374] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.235998 14674 raft_consensus.cc:348] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.236681 14675 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.236824 14674 raft_consensus.cc:374] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.237457 14675 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.238194 14676 tablet_bootstrap.cc:492] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:29.235868 14755 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 20:34:29.238787 14676 ts_tablet_manager.cc:1397] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.065s user 0.026s sys 0.026s
I20260806 20:34:29.240057 14478 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:29.237089 14674 raft_consensus.cc:729] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.241360 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:29.241629 14674 consensus_queue.cc:260] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.241822 14676 raft_consensus.cc:348] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.242192 14681 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:29.242398 14676 raft_consensus.cc:374] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.242738 14681 raft_consensus.cc:686] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:29.242803 14676 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.243639 14925 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:29.244115 14925 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:29.243611 14681 consensus_queue.cc:237] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.243639 14676 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.244421 14925 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:29.248539 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:29.248771 14676 ts_tablet_manager.cc:1428] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260806 20:34:29.249933 14676 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:29.254750 14756 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.119552 14372 catalog_manager.cc:5495] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:29.255198 14756 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.257164 14756 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:29.257688 14841 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.258152 14841 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.258662 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:29.259306 14477 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:29.245656 14675 ts_tablet_manager.cc:1428] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20260806 20:34:29.260051 14841 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.260479 14595 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:29.260921 14675 tablet_bootstrap.cc:492] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:29.261404 14756 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:29.261768 14756 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:29.245376 14674 ts_tablet_manager.cc:1428] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260806 20:34:29.261884 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:29.262109 14756 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.263619 14674 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:29.259831 14369 catalog_manager.cc:5495] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:29.265208 14676 tablet_bootstrap.cc:654] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.265645 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:29.266366 14444 leader_election.cc:341] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.254092 14925 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.228197 14555 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:29.276971 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:29.277628 14553 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
W20260806 20:34:29.278697 14442 leader_election.cc:341] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.279131 14442 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:29.279884 14841 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:29.284570 14675 tablet_bootstrap.cc:654] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.285567 14925 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.286576 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:29.287268 14628 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.292310 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:29.294227 14555 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.296216 14674 tablet_bootstrap.cc:654] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.300048 14756 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.307350 14628 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:29.313596 14444 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:29.314909 14841 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:29.317576 14841 raft_consensus.cc:686] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:29.318771 14841 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.324079 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:29.324790 14478 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.328151 14555 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:29.325505 14756 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:29.328662 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:29.329511 14553 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.335848 14553 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:29.336916 14596 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:29.337935 14756 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:29.339540 14756 raft_consensus.cc:686] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:29.341041 14756 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.342844 14478 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:29.341804 14676 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:29.344131 14676 ts_tablet_manager.cc:1397] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.094s user 0.021s sys 0.004s
I20260806 20:34:29.348059 14675 tablet_bootstrap.cc:492] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:29.348562 14675 ts_tablet_manager.cc:1397] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.004s
I20260806 20:34:29.351302 14676 raft_consensus.cc:348] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.352136 14676 raft_consensus.cc:374] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.352051 14755 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:29.352679 14676 raft_consensus.cc:729] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.353394 14676 consensus_queue.cc:260] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.355259 14477 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.356163 14675 raft_consensus.cc:348] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.356907 14675 raft_consensus.cc:374] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.357172 14675 raft_consensus.cc:729] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.358455 14676 ts_tablet_manager.cc:1428] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 20:34:29.359928 14676 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:29.360478 14675 consensus_queue.cc:260] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.371729 14755 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 20:34:29.377795 14675 ts_tablet_manager.cc:1428] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260806 20:34:29.378715 14675 tablet_bootstrap.cc:492] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:29.392911 14372 catalog_manager.cc:5495] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:29.402159 14676 tablet_bootstrap.cc:654] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.413687 14755 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.414270 14755 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.421011 14755 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:29.423218 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:29.423763 14675 tablet_bootstrap.cc:654] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.425171 14755 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:29.431903 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:29.451732 14595 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:29.444887 14826 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.458179 14826 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.453894 14756 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:29.460131 14826 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.466538 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:29.460841 14628 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.462556 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:29.473776 14521 leader_election.cc:341] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.475617 14756 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:34:29.482016 14518 leader_election.cc:341] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.482502 14518 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:29.483807 14761 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 20:34:29.546669 15008 log.cc:924] Time spent T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260806 20:34:29.548620 14674 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:29.549134 14674 ts_tablet_manager.cc:1397] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.286s user 0.042s sys 0.040s
W20260806 20:34:29.554409 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 70237.4us
I20260806 20:34:29.575902 14372 catalog_manager.cc:5495] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:29.552070 14674 raft_consensus.cc:348] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.577662 14676 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:29.580202 14675 tablet_bootstrap.cc:492] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:29.581508 14675 ts_tablet_manager.cc:1397] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.203s user 0.020s sys 0.007s
I20260806 20:34:29.581508 14676 ts_tablet_manager.cc:1397] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.222s user 0.012s sys 0.017s
I20260806 20:34:29.584125 14675 raft_consensus.cc:348] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.591593 14675 raft_consensus.cc:374] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.591924 14675 raft_consensus.cc:729] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.592552 14675 consensus_queue.cc:260] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.593686 14841 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.594129 14841 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.594714 14675 ts_tablet_manager.cc:1428] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260806 20:34:29.595588 14675 tablet_bootstrap.cc:492] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
W20260806 20:34:29.597129 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 111280us
I20260806 20:34:29.599511 14676 raft_consensus.cc:348] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.600157 14676 raft_consensus.cc:374] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.600407 14676 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.601020 14841 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.601401 14676 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.607139 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:29.581871 14674 raft_consensus.cc:374] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 20:34:29.608597 14444 leader_election.cc:341] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.608762 14674 raft_consensus.cc:729] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.609993 14674 consensus_queue.cc:260] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.627101 14674 ts_tablet_manager.cc:1428] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
I20260806 20:34:29.630028 14676 ts_tablet_manager.cc:1428] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.048s user 0.004s sys 0.000s
I20260806 20:34:29.633456 14676 tablet_bootstrap.cc:492] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:29.634599 14674 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:29.639118 14676 tablet_bootstrap.cc:654] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.639178 14675 tablet_bootstrap.cc:654] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.653715 14756 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260806 20:34:29.665699 15011 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.666152 15011 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.673475 14841 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:29.656931 14674 tablet_bootstrap.cc:654] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:29.657297 14707 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260806 20:34:29.649729 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:29.681052 14552 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:29.687700 14442 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:29.689365 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:29.691107 14841 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:29.691316 14877 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 20:34:29.692754 14518 leader_election.cc:341] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.691711 14841 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:29.694015 14841 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.745992 14878 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.753892 14878 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.737217 14681 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.737546 15011 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.821647 15011 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.823383 15011 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.824023 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:29.824909 14552 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:29.828907 14878 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.831393 14707 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:29.831821 14707 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.834421 14707 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:29.797292 14681 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.834857 14841 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.842053 14675 tablet_bootstrap.cc:492] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:29.842741 14675 ts_tablet_manager.cc:1397] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.247s user 0.018s sys 0.027s
I20260806 20:34:29.834450 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:29.826408 15011 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.848955 14878 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:34:29.854336 14674 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:29.854815 14674 ts_tablet_manager.cc:1397] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.221s user 0.012s sys 0.015s
I20260806 20:34:29.856164 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:29.858333 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:29.859287 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:29.857396 14521 leader_election.cc:341] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.851678 14878 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:29.862560 14878 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:29.863860 14674 raft_consensus.cc:348] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.864732 14674 raft_consensus.cc:374] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.866840 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:29.867345 14707 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:29.866878 14674 raft_consensus.cc:729] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.870337 14674 consensus_queue.cc:260] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.870726 14841 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:29.872553 14521 leader_election.cc:341] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:29.872942 14521 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:29.885097 15010 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:29.886636 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:29.887435 14630 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:29.887950 14878 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.889894 14878 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:29.892676 14676 tablet_bootstrap.cc:492] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:29.893111 14676 ts_tablet_manager.cc:1397] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.260s user 0.010s sys 0.020s
I20260806 20:34:29.895862 14676 raft_consensus.cc:348] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.896637 14676 raft_consensus.cc:374] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:29.896934 14676 raft_consensus.cc:729] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:29.889969 14552 raft_consensus.cc:2396] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 20:34:29.897532 14676 consensus_queue.cc:260] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:29.898530 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:29.901046 14554 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:29.899847 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:29.906035 14674 ts_tablet_manager.cc:1428] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.051s user 0.006s sys 0.002s
I20260806 20:34:29.900077 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:29.915236 14674 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:29.915083 14554 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:29.947121 14550 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:29.952386 14475 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:30.033411 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:30.034454 14628 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:30.039294 14675 raft_consensus.cc:348] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.042171 14675 raft_consensus.cc:374] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.112811 14675 raft_consensus.cc:729] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.113749 14675 consensus_queue.cc:260] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.051846 14676 ts_tablet_manager.cc:1428] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.158s user 0.005s sys 0.002s
I20260806 20:34:30.055619 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:30.062305 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:30.069403 14755 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:34:30.075730 14444 leader_election.cc:341] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:30.124675 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 48981.8us
I20260806 20:34:30.078325 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.125689 14676 tablet_bootstrap.cc:492] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:30.084111 14550 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:34:30.102576 14681 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:30.103799 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:30.105993 15011 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:30.082295 14628 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:34:30.126282 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 53395.9us
I20260806 20:34:30.132339 14549 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:30.133323 15011 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.135402 15011 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.136584 14596 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:30.138537 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:30.138971 15012 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:30.139567 15012 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:30.140163 15012 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.149044 14444 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:30.150740 14878 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:30.151429 14878 raft_consensus.cc:686] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:30.150483 14681 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:30.152194 14681 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:30.152314 14877 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:30.153014 14681 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.152936 14878 consensus_queue.cc:237] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.157866 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:30.166221 14518 leader_election.cc:341] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:30.168099 14521 leader_election.cc:341] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.168541 14521 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.172638 14674 tablet_bootstrap.cc:654] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:30.175726 15011 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:30.177327 14442 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.180442 14675 ts_tablet_manager.cc:1428] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.337s user 0.006s sys 0.003s
I20260806 20:34:30.181756 14675 tablet_bootstrap.cc:492] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:30.182271 14877 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:34:30.178819 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 79556.4us
I20260806 20:34:30.189707 15010 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:30.190338 15010 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.193114 15020 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:30.195295 15010 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.195649 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:30.208528 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:30.209156 14632 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:30.210193 14841 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:34:30.210683 14841 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.218199 15020 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:34:30.218151 14681 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.221772 14681 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:30.212656 14841 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.183593 14877 raft_consensus.cc:686] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:30.235190 14675 tablet_bootstrap.cc:654] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:30.238727 14595 leader_election.cc:341] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.243880 14676 tablet_bootstrap.cc:654] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:30.246729 15012 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.251703 14521 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:30.253382 14877 consensus_queue.cc:237] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.256124 15011 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:30.256881 15011 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:30.257599 15011 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.237048 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:30.237987 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:30.269532 14478 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:30.274315 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:30.279623 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:30.289052 14633 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 2: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:30.294072 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:30.297891 15012 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:30.300714 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:30.280908 14549 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.312650 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:34:30.315735 14595 leader_election.cc:341] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:30.316541 14521 leader_election.cc:341] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.300648 14475 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.328349 14676 tablet_bootstrap.cc:492] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:30.328971 14676 ts_tablet_manager.cc:1397] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.203s user 0.017s sys 0.027s
I20260806 20:34:30.358134 14925 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:30.404982 14826 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:30.413833 14826 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.417114 14676 raft_consensus.cc:348] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.417832 14676 raft_consensus.cc:374] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.418069 14676 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.418623 14676 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:30.421514 15028 log.cc:924] Time spent T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.092s user 0.002s sys 0.000s
I20260806 20:34:30.411026 14925 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.396368 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:30.433688 14676 ts_tablet_manager.cc:1428] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.104s user 0.007s sys 0.001s
I20260806 20:34:30.434580 14676 tablet_bootstrap.cc:492] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:30.428125 15011 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.465615 14675 tablet_bootstrap.cc:492] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:30.476554 14675 ts_tablet_manager.cc:1397] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.295s user 0.020s sys 0.021s
I20260806 20:34:30.451186 14550 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:34:30.470415 14925 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.479074 14674 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
W20260806 20:34:30.478998 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 46201.6us
I20260806 20:34:30.479600 14674 ts_tablet_manager.cc:1397] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.565s user 0.038s sys 0.050s
I20260806 20:34:30.495767 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:30.496485 14628 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:34:30.497536 14521 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:30.498355 14683 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:34:30.498682 14683 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:30.499015 14683 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:30.496440 14826 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.511466 14683 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.513450 14683 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.517198 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:30.518081 14476 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:30.516532 14674 raft_consensus.cc:348] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.520227 14674 raft_consensus.cc:374] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.520480 14674 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.521286 14674 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.521544 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:30.521962 15011 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.522562 14633 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:34:30.526191 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:30.529660 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:30.530364 14476 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:34:30.514251 14675 raft_consensus.cc:348] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.531679 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:30.531862 14675 raft_consensus.cc:374] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.532222 14675 raft_consensus.cc:729] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.532261 14628 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.537879 14674 ts_tablet_manager.cc:1428] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.058s user 0.009s sys 0.000s
I20260806 20:34:30.539192 14674 tablet_bootstrap.cc:492] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:30.540519 14675 consensus_queue.cc:260] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.543182 14628 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:30.544023 14521 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:30.546280 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:30.547154 14474 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.556295 14683 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:30.556829 14683 raft_consensus.cc:686] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:30.557459 14683 consensus_queue.cc:237] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.555459 14475 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:30.571699 14675 ts_tablet_manager.cc:1428] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.095s user 0.010s sys 0.004s
I20260806 20:34:30.573559 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:30.578683 14555 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:30.582957 14676 tablet_bootstrap.cc:654] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:30.584797 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.585536 14675 tablet_bootstrap.cc:492] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:30.587874 15010 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:30.588562 15010 raft_consensus.cc:686] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:34:30.596406 14444 leader_election.cc:341] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.602190 15010 consensus_queue.cc:237] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.621376 14674 tablet_bootstrap.cc:654] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:30.624742 14631 raft_consensus.cc:1261] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 20:34:30.628659 14633 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:34:30.627806 15032 log.cc:924] Time spent T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.141s user 0.000s sys 0.006s
I20260806 20:34:30.636613 14370 catalog_manager.cc:5495] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:30.643466 14370 catalog_manager.cc:5495] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 2, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:30.646291 15036 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:30.649118 14521 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:30.650532 14683 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:34:30.640348 14925 consensus_queue.cc:1035] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260806 20:34:30.651609 14476 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:34:30.651903 14442 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.653282 14683 raft_consensus.cc:686] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:30.654958 15035 raft_consensus.cc:2741] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 20:34:30.657848 14683 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.658339 14442 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.660180 15035 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:30.660984 15035 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:30.661753 15035 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.673731 14676 tablet_bootstrap.cc:492] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:30.674306 14676 ts_tablet_manager.cc:1397] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.240s user 0.012s sys 0.017s
I20260806 20:34:30.645716 14549 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:30.675886 14596 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:30.677295 14755 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:30.678910 14755 raft_consensus.cc:686] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:30.679934 14755 consensus_queue.cc:237] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.702150 14676 raft_consensus.cc:348] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.703289 14676 raft_consensus.cc:374] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.703614 14676 raft_consensus.cc:729] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.704437 14676 consensus_queue.cc:260] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.711743 14676 ts_tablet_manager.cc:1428] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.037s user 0.006s sys 0.002s
I20260806 20:34:30.712590 14676 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:30.713042 15036 consensus_queue.cc:1035] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260806 20:34:30.739734 15035 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.747859 14675 tablet_bootstrap.cc:654] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
W20260806 20:34:30.757835 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 61799.1us
I20260806 20:34:30.783843 14676 tablet_bootstrap.cc:654] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:30.788820 14474 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:30.791311 14674 tablet_bootstrap.cc:492] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:30.791870 14674 ts_tablet_manager.cc:1397] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.253s user 0.009s sys 0.013s
I20260806 20:34:30.793713 14755 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:30.795188 15035 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.794999 14755 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.797621 14369 catalog_manager.cc:5495] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:30.798831 14369 catalog_manager.cc:5495] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:30.801266 14552 raft_consensus.cc:1261] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 20:34:30.808434 15035 consensus_queue.cc:1035] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:30.805886 14371 catalog_manager.cc:5495] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:30.814004 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:34:30.814133 14444 consensus_peers.cc:487] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 -> Peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Couldn't send request to peer df1245ecd2034e11ae6cf688c1c10a4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 20:34:30.815131 14444 leader_election.cc:341] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.815955 14755 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:30.815992 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:30.823616 14549 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.843786 14675 tablet_bootstrap.cc:492] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:30.844556 14675 ts_tablet_manager.cc:1397] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.272s user 0.023s sys 0.014s
I20260806 20:34:30.844398 14674 raft_consensus.cc:348] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.845376 14674 raft_consensus.cc:374] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.845651 14674 raft_consensus.cc:729] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.842361 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:30.846247 14674 consensus_queue.cc:260] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.848524 14552 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:30.850915 14596 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:30.853762 14674 ts_tablet_manager.cc:1428] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.062s user 0.006s sys 0.007s
I20260806 20:34:30.854725 14674 tablet_bootstrap.cc:492] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:30.856359 14549 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:30.858561 15050 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:30.859018 15050 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:30.859377 15050 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.863534 14442 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.864297 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:30.864995 15036 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:30.865507 15036 raft_consensus.cc:686] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:30.866415 15036 consensus_queue.cc:237] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.876706 14675 raft_consensus.cc:348] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.877365 15010 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:30.877969 15010 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.880398 15010 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:30.884018 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:30.887355 14518 leader_election.cc:341] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.887450 14674 tablet_bootstrap.cc:654] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:30.881798 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:30.877514 14675 raft_consensus.cc:374] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.891420 14675 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.896724 14675 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:30.901088 14521 leader_election.cc:341] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.901713 14521 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:30.903795 14675 ts_tablet_manager.cc:1428] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.059s user 0.007s sys 0.002s
I20260806 20:34:30.917888 14675 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:30.921249 15037 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:30.922766 14676 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:30.923264 14676 ts_tablet_manager.cc:1397] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.211s user 0.024s sys 0.007s
I20260806 20:34:30.928694 15050 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.928373 14676 raft_consensus.cc:348] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.929553 14676 raft_consensus.cc:374] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:30.929853 14676 raft_consensus.cc:729] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:30.981954 14676 consensus_queue.cc:260] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:30.986354 15050 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:30.988287 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:30.989020 14553 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:30.993826 14675 tablet_bootstrap.cc:654] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:30.996196 14676 ts_tablet_manager.cc:1428] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.073s user 0.006s sys 0.001s
I20260806 20:34:30.997107 14676 tablet_bootstrap.cc:492] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
I20260806 20:34:30.997951 14553 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:30.997860 14745 consensus_queue.cc:1035] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:30.999104 14596 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:34:31.002453 14595 leader_election.cc:341] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:30.999979 15054 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:31.004868 15054 raft_consensus.cc:686] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:31.006215 15054 consensus_queue.cc:237] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.010118 14745 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.010591 14745 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.013677 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:31.012375 14745 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:31.016372 15036 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.016868 15036 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.017405 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:31.018570 14521 leader_election.cc:341] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.030503 15036 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:31.031818 14676 tablet_bootstrap.cc:654] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:31.032297 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:31.019114 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:31.034610 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:34:31.036782 14442 leader_election.cc:341] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.035022 14478 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:31.039055 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.040496 14745 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:31.040861 14745 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:31.041213 14745 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.054054 15047 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.054916 15047 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.063405 14745 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:31.065784 14444 leader_election.cc:341] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.066220 14444 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.066566 14745 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:31.067262 14924 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:31.071352 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:34:31.072988 14521 leader_election.cc:341] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.074913 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:31.075855 14473 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.082433 15012 consensus_queue.cc:1035] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:31.095788 15047 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:31.097607 14628 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:34:31.097270 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:31.098603 14555 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:31.100490 14596 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:34:31.103732 14595 leader_election.cc:341] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.104516 15050 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:31.105214 15050 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:31.105871 15050 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.108279 14674 tablet_bootstrap.cc:492] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:31.110078 14674 ts_tablet_manager.cc:1397] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.256s user 0.029s sys 0.020s
I20260806 20:34:31.111222 15037 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.016s
W20260806 20:34:31.145716 14521 consensus_peers.cc:487] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Couldn't send request to peer df1245ecd2034e11ae6cf688c1c10a4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:31.157282 14478 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:34:31.184688 14745 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260806 20:34:31.185797 14370 catalog_manager.cc:5495] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:31.184587 14674 raft_consensus.cc:348] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.201002 14674 raft_consensus.cc:374] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.201256 14674 raft_consensus.cc:729] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:31.201650 14674 consensus_queue.cc:260] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.209529 14473 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:31.217262 15050 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.221279 15050 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:31.229233 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.233810 14745 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:31.234426 14745 raft_consensus.cc:686] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:31.235095 14745 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.241952 14674 ts_tablet_manager.cc:1428] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.132s user 0.011s sys 0.001s
I20260806 20:34:31.244118 14674 tablet_bootstrap.cc:492] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:31.250813 15047 consensus_queue.cc:1035] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:31.260761 14675 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:31.247031 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:31.251436 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:31.240628 15037 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260806 20:34:31.264768 14676 tablet_bootstrap.cc:492] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:31.261277 14675 ts_tablet_manager.cc:1397] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.344s user 0.035s sys 0.020s
I20260806 20:34:31.348516 14676 ts_tablet_manager.cc:1397] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.352s user 0.029s sys 0.043s
W20260806 20:34:31.345525 15069 log.cc:924] Time spent T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260806 20:34:31.343600 15012 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.371668 14555 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.374853 15012 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.411088 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:31.411361 15012 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:31.415496 14555 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:31.418453 14596 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:31.431219 14676 raft_consensus.cc:348] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.431803 14676 raft_consensus.cc:374] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.432080 14676 raft_consensus.cc:729] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
I20260806 20:34:31.432698 14676 consensus_queue.cc:260] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.433727 14675 raft_consensus.cc:348] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.434264 14675 raft_consensus.cc:374] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.434497 14675 raft_consensus.cc:729] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:31.441722 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:31.421845 15020 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:31.445591 14675 consensus_queue.cc:260] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:31.447798 14518 leader_election.cc:341] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:34:31.454967 14595 leader_election.cc:341] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.450129 14371 catalog_manager.cc:5495] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
W20260806 20:34:31.460391 14595 consensus_peers.cc:487] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Couldn't send request to peer 92828dd6dbe74f29a04d4e702b502621. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:31.462077 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:31.459623 15020 raft_consensus.cc:686] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:31.466441 14674 tablet_bootstrap.cc:654] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:31.460529 14676 ts_tablet_manager.cc:1428] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.112s user 0.002s sys 0.006s
I20260806 20:34:31.468422 14676 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: Bootstrap starting.
W20260806 20:34:31.468609 14595 leader_election.cc:341] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.466555 15020 consensus_queue.cc:237] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.486161 14878 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.487004 14878 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.494463 15036 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:31.496184 14878 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:31.506906 14521 leader_election.cc:341] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.509658 14521 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.520216 15075 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:31.525317 14675 ts_tablet_manager.cc:1428] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.177s user 0.003s sys 0.005s
I20260806 20:34:31.526454 14675 tablet_bootstrap.cc:492] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:31.528648 14369 catalog_manager.cc:5495] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:31.555004 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:31.533798 14826 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.569726 14675 tablet_bootstrap.cc:654] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:31.564895 14551 raft_consensus.cc:2379] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 1: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:34:31.577680 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:31.558408 14826 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.622964 14826 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:31.632087 14676 tablet_bootstrap.cc:654] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:31.632764 14674 tablet_bootstrap.cc:492] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:31.633245 14674 ts_tablet_manager.cc:1397] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.390s user 0.017s sys 0.022s
W20260806 20:34:31.643115 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47726.4us
I20260806 20:34:31.655028 14674 raft_consensus.cc:348] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.655695 14674 raft_consensus.cc:374] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.655958 14674 raft_consensus.cc:729] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:31.656490 14674 consensus_queue.cc:260] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.670121 14626 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.669977 14442 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.684377 14674 ts_tablet_manager.cc:1428] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20260806 20:34:31.685367 14674 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:31.695039 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:31.702275 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47239.9us
W20260806 20:34:31.707185 14521 leader_election.cc:341] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.688467 14476 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:31.718207 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 57852.4us
I20260806 20:34:31.736356 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:31.739163 14470 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
W20260806 20:34:31.768850 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:31.781437 14675 tablet_bootstrap.cc:492] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:31.781270 14518 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.781888 14675 ts_tablet_manager.cc:1397] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.256s user 0.021s sys 0.026s
I20260806 20:34:31.780740 14369 catalog_manager.cc:5495] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:31.795418 14675 raft_consensus.cc:348] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.796152 14675 raft_consensus.cc:374] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.801081 15012 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:31.801523 15012 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:31.801862 15012 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.796411 14675 raft_consensus.cc:729] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:31.809886 15012 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.810024 14675 consensus_queue.cc:260] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.811729 14675 ts_tablet_manager.cc:1428] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20260806 20:34:31.814191 14924 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:31.812826 14674 tablet_bootstrap.cc:654] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:31.814352 14675 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:31.818111 15050 consensus_queue.cc:1035] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:31.819586 15047 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.820031 15047 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.821943 15047 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:31.823885 15086 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:31.831322 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:34:31.832863 14521 leader_election.cc:341] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.833033 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:31.834729 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:31.835572 14476 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.824918 15012 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:31.856495 14676 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: No bootstrap required, opened a new log
I20260806 20:34:31.857255 14676 ts_tablet_manager.cc:1397] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent bootstrapping tablet: real 0.389s user 0.027s sys 0.047s
I20260806 20:34:31.865541 14476 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:31.867311 14518 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.868819 15012 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:31.869329 15012 raft_consensus.cc:686] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:31.870035 15012 consensus_queue.cc:237] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.871167 14675 tablet_bootstrap.cc:654] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:31.876614 15078 consensus_queue.cc:1035] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260806 20:34:31.900862 15090 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 20:34:31.896754 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:31.908314 14676 raft_consensus.cc:348] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.909385 14676 raft_consensus.cc:374] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.909685 14676 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Initialized, Role: FOLLOWER
W20260806 20:34:31.910342 14595 consensus_peers.cc:487] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Couldn't send request to peer 92828dd6dbe74f29a04d4e702b502621. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:31.911517 14676 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.914665 14471 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:31.914692 14676 ts_tablet_manager.cc:1428] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent starting tablet: real 0.057s user 0.005s sys 0.001s
I20260806 20:34:31.913889 14674 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:31.918859 14674 ts_tablet_manager.cc:1397] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.234s user 0.019s sys 0.004s
I20260806 20:34:31.918220 14370 catalog_manager.cc:5495] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:31.921026 14674 raft_consensus.cc:348] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:31.921955 14521 consensus_peers.cc:487] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Couldn't send request to peer df1245ecd2034e11ae6cf688c1c10a4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:31.922025 14674 raft_consensus.cc:374] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.922741 14674 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:31.925325 14674 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.927912 14596 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:31.931641 15078 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:34:31.935501 14674 ts_tablet_manager.cc:1428] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.016s user 0.003s sys 0.005s
I20260806 20:34:31.936793 14674 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:31.944376 14674 tablet_bootstrap.cc:654] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:31.947096 15012 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:31.947611 15012 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.950150 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:31.951516 14518 leader_election.cc:341] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.953366 15012 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:31.954335 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:31.955755 14521 leader_election.cc:341] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:31.956130 14521 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:31.957158 15012 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:31.959888 14675 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:31.960426 14675 ts_tablet_manager.cc:1397] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.146s user 0.012s sys 0.009s
I20260806 20:34:31.963528 14675 raft_consensus.cc:348] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.964409 14675 raft_consensus.cc:374] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:31.965196 14675 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:31.966197 14675 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:31.968793 14675 ts_tablet_manager.cc:1428] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260806 20:34:31.971230 14675 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:31.972636 14370 catalog_manager.cc:5495] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:31.997211 14674 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:31.997799 14674 ts_tablet_manager.cc:1397] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.011s
I20260806 20:34:32.003285 14675 tablet_bootstrap.cc:654] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.001991 14674 raft_consensus.cc:348] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.011159 14674 raft_consensus.cc:374] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.011436 14674 raft_consensus.cc:729] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.012185 14674 consensus_queue.cc:260] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.014525 14674 ts_tablet_manager.cc:1428] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.016s user 0.007s sys 0.002s
I20260806 20:34:32.015591 14674 tablet_bootstrap.cc:492] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:32.025712 14675 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:32.026226 14675 ts_tablet_manager.cc:1397] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.005s
I20260806 20:34:32.031562 14675 raft_consensus.cc:348] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.032564 14675 raft_consensus.cc:374] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.032931 14675 raft_consensus.cc:729] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.035266 14674 tablet_bootstrap.cc:654] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.038034 14675 consensus_queue.cc:260] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.038466 15078 consensus_queue.cc:1035] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:32.040944 14675 ts_tablet_manager.cc:1428] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260806 20:34:32.042454 14675 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:32.047186 15012 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:32.080071 15012 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.054338 14675 tablet_bootstrap.cc:654] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.072132 15050 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:32.088096 15050 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.093305 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:32.090554 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:32.096474 14471 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:32.098358 14518 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:32.100827 15088 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:32.099792 15012 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:32.101614 15088 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:32.102612 15088 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.104391 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:32.105687 14633 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:32.106786 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:32.110242 14595 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:32.111152 15078 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:32.112639 15050 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:32.135334 14674 tablet_bootstrap.cc:492] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:32.135984 14674 ts_tablet_manager.cc:1397] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.121s user 0.025s sys 0.039s
I20260806 20:34:32.135202 15050 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:32.147286 15050 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.151966 15088 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.164130 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:32.165189 14471 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.165724 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:32.166527 14627 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.169688 15088 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:32.169885 14674 raft_consensus.cc:348] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.181995 14674 raft_consensus.cc:374] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.182293 14674 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.185250 15035 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:32.185930 14675 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:32.186380 14675 ts_tablet_manager.cc:1397] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.145s user 0.011s sys 0.010s
I20260806 20:34:32.186124 15035 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.193317 15035 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:32.241844 15050 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:32.243485 14471 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:32.254225 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:32.254868 14627 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:32.256992 14518 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:32.258159 14745 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:32.258714 14745 raft_consensus.cc:686] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:32.259521 14745 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.260987 14675 raft_consensus.cc:348] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.261881 14675 raft_consensus.cc:374] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.262217 14675 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.246073 14674 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.248801 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:34:32.252245 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 66969.3us
W20260806 20:34:32.245832 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 51752.9us
I20260806 20:34:32.263029 14675 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.270888 14674 ts_tablet_manager.cc:1428] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.135s user 0.002s sys 0.009s
I20260806 20:34:32.271713 14674 tablet_bootstrap.cc:492] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:32.278759 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:32.279422 14551 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 0.
I20260806 20:34:32.281067 14675 ts_tablet_manager.cc:1428] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.094s user 0.011s sys 0.011s
I20260806 20:34:32.282047 14675 tablet_bootstrap.cc:492] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
W20260806 20:34:32.285531 14444 leader_election.cc:341] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:32.299521 14442 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:32.303120 15035 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:32.303905 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:32.307286 14595 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:32.308401 15047 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:32.309490 14675 tablet_bootstrap.cc:654] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.304637 15035 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:32.312325 15035 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.316744 14674 tablet_bootstrap.cc:654] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.372087 15035 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.389252 14925 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:32.391566 14925 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.394981 14925 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:32.429399 14627 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.429968 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:32.433249 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:32.434360 14629 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.417580 14626 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:32.452170 14521 consensus_peers.cc:487] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Couldn't send request to peer df1245ecd2034e11ae6cf688c1c10a4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:32.455647 14442 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:32.468040 15090 consensus_queue.cc:1035] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:32.492672 14675 tablet_bootstrap.cc:492] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:32.493799 14675 ts_tablet_manager.cc:1397] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.212s user 0.012s sys 0.010s
I20260806 20:34:32.497151 15035 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:32.512780 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 57448.3us
I20260806 20:34:32.513316 14369 catalog_manager.cc:5495] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:32.515416 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:32.516505 14551 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:34:32.530304 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 96486.7us
I20260806 20:34:32.525905 15035 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:32.541616 14674 tablet_bootstrap.cc:492] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:32.542994 14674 ts_tablet_manager.cc:1397] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.271s user 0.006s sys 0.014s
W20260806 20:34:32.562778 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 56412.5us
I20260806 20:34:32.570034 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:32.594233 14675 raft_consensus.cc:348] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.607826 14471 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.608492 14674 raft_consensus.cc:348] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:32.585510 14444 leader_election.cc:341] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:34:32.613269 14674 raft_consensus.cc:374] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.618366 14674 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.613721 14675 raft_consensus.cc:374] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.620339 14675 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.619433 14674 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.614461 15089 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:32.625474 15089 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.630080 15089 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:32.628153 14675 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.632198 14675 ts_tablet_manager.cc:1428] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.138s user 0.002s sys 0.007s
I20260806 20:34:32.633176 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:32.633540 14675 tablet_bootstrap.cc:492] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:32.633860 14479 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:32.635506 14518 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:32.640830 15090 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:32.641229 15090 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:32.641649 15090 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.618101 14551 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:34:32.651383 15090 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.657055 14442 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:32.659235 14674 ts_tablet_manager.cc:1428] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.111s user 0.007s sys 0.003s
I20260806 20:34:32.660974 14924 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:32.661332 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:32.661556 14924 raft_consensus.cc:686] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:32.662066 14629 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 0.
I20260806 20:34:32.662561 14924 consensus_queue.cc:237] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.664100 14675 tablet_bootstrap.cc:654] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.689994 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:32.690694 14476 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.697219 15090 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:32.698519 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:32.699343 14629 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.700366 14476 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:32.702606 14674 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Bootstrap starting.
I20260806 20:34:32.727785 14629 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:34:32.728827 14518 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:32.729930 15088 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:32.730391 15088 raft_consensus.cc:686] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:32.748090 15088 consensus_queue.cc:237] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.748678 15077 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:34:32.749243 15077 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.757331 15077 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:32.759158 14674 tablet_bootstrap.cc:654] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.760154 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:32.765962 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:32.777017 14596 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:32.782938 15077 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:34:32.800484 15090 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:32.802604 15075 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:32.807592 14675 tablet_bootstrap.cc:492] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:32.808014 14675 ts_tablet_manager.cc:1397] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.175s user 0.011s sys 0.014s
I20260806 20:34:32.810438 14675 raft_consensus.cc:348] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.811153 14675 raft_consensus.cc:374] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.811395 14675 raft_consensus.cc:729] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.810940 14371 catalog_manager.cc:5495] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:32.811991 14675 consensus_queue.cc:260] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.814760 14675 ts_tablet_manager.cc:1428] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 20:34:32.815833 14675 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:32.825098 14675 tablet_bootstrap.cc:654] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.840992 14674 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: No bootstrap required, opened a new log
I20260806 20:34:32.841447 14674 ts_tablet_manager.cc:1397] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Time spent bootstrapping tablet: real 0.175s user 0.026s sys 0.012s
I20260806 20:34:32.843940 14674 raft_consensus.cc:348] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.845131 14674 raft_consensus.cc:374] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.845407 14674 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.846001 14674 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.849609 14674 ts_tablet_manager.cc:1428] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:34:32.861142 14675 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:32.861649 14675 ts_tablet_manager.cc:1397] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.000s
I20260806 20:34:32.863910 14675 raft_consensus.cc:348] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.864512 14675 raft_consensus.cc:374] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.864749 14675 raft_consensus.cc:729] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.870510 14675 consensus_queue.cc:260] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.871793 15047 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:32.872291 15047 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.872670 14675 ts_tablet_manager.cc:1428] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260806 20:34:32.874956 14675 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Bootstrap starting.
I20260806 20:34:32.875156 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:32.878207 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:32.879016 14552 raft_consensus.cc:2379] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 1: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:34:32.880088 14596 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:32.880976 15077 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:32.884363 14675 tablet_bootstrap.cc:654] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Neither blocks nor log segments found. Creating new log.
I20260806 20:34:32.872656 14369 catalog_manager.cc:5495] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:32.892052 15047 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:32.909430 15077 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:34:32.911166 14675 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: No bootstrap required, opened a new log
I20260806 20:34:32.911645 14675 ts_tablet_manager.cc:1397] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.005s
W20260806 20:34:32.912712 14421 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:34:32.926539 14675 raft_consensus.cc:348] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.927278 14675 raft_consensus.cc:374] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:34:32.927542 14675 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Initialized, Role: FOLLOWER
I20260806 20:34:32.928254 14675 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:32.932250 14675 ts_tablet_manager.cc:1428] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent starting tablet: real 0.020s user 0.011s sys 0.003s
I20260806 20:34:32.999752 15077 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:33.000257 15077 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:33.002058 15077 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:33.002704 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:34:33.003300 14476 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:33.004402 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:33.005012 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:33.005224 15077 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:34:33.005638 15077 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:34:33.005681 14549 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 0.
I20260806 20:34:33.006052 15077 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:33.012868 15077 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:33.014787 15077 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:33.015291 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:33.015904 14478 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:33.016088 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:33.016678 14549 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:33.022957 14478 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:33.023132 14549 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:34:33.024077 14596 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:33.024818 15077 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:34:33.025318 15077 raft_consensus.cc:686] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:33.026203 15077 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:33.034291 14629 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:34:33.041399 14371 catalog_manager.cc:5495] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:34:33.099758 14924 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:33.115483 14925 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:33.134778 15124 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:34:33.164618 15088 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:33.495579 15047 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:33.507926 15110 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:34:34.686498 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:34:34.694471 15141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:34.714807 15144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:34:34.714855 15142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:34:34.717020 13282 server_base.cc:1034] running on GCE node
I20260806 20:34:34.718938 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:34:34.719190 13282 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:34:34.719415 13282 hybrid_clock.cc:648] HybridClock initialized: now 1786048474719398 us; error 0 us; skew 500 ppm
I20260806 20:34:34.720064 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:34:34.729862 13282 webserver.cc:460] Webserver started at http://127.12.248.132:38691/ using document root <none> and password file <none>
I20260806 20:34:34.730386 13282 fs_manager.cc:359] Metadata directory not provided
I20260806 20:34:34.730607 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:34:34.730952 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:34:34.732235 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-3-root/instance:
uuid: "148bf2a11dba423ab364371f0458e871"
format_stamp: "Formatted at 2026-08-06 20:34:34 on dist-test-slave-10pc"
I20260806 20:34:34.737862 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 20:34:34.741811 15149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:34.742816 13282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:34:34.743486 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-3-root
uuid: "148bf2a11dba423ab364371f0458e871"
format_stamp: "Formatted at 2026-08-06 20:34:34 on dist-test-slave-10pc"
I20260806 20:34:34.743866 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:34:34.763751 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:34:34.765259 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:34:34.766727 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:34:34.769098 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:34:34.769309 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:34.769596 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:34:34.769759 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:34:34.830493 13282 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.132:42399
I20260806 20:34:34.830664 15212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.132:42399 every 8 connection(s)
I20260806 20:34:34.835322 13282 auto_leader_rebalancer-test.cc:413] add a tserver: 148bf2a11dba423ab364371f0458e871
I20260806 20:34:34.850992 15213 heartbeater.cc:347] Connected to a master server at 127.12.248.190:39439
I20260806 20:34:34.851480 15213 heartbeater.cc:464] Registering TS with master...
I20260806 20:34:34.852658 15213 heartbeater.cc:511] Master 127.12.248.190:39439 requested a full tablet report, sending...
I20260806 20:34:34.854928 14370 ts_manager.cc:194] Registered new tserver with Master: 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:34:34.856974 14370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56582
I20260806 20:34:35.036152 14471 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.043365 14549 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.043763 14626 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.044895 14924 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:35.046489 15140 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:35.072188 15140 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.076514 14550 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
W20260806 20:34:35.080356 14441 consensus_peers.cc:487] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f38960b1dd9245a8ba85fea601b09e94. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:35.082935 14626 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.095429 14370 catalog_manager.cc:5495] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.098620 14476 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
W20260806 20:34:35.122093 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 49611.1us
I20260806 20:34:35.170868 14626 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.172531 15221 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 20:34:35.176643 14441 consensus_peers.cc:487] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a53ea52d2ecf44bd87a1c23ea5f5f8df. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:35.179489 14549 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.180989 15140 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:35.185644 15140 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.188212 14629 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.211212 14549 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.229417 14369 catalog_manager.cc:5495] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:35.241668 14626 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.249897 14550 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.251885 15020 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:35.252801 14476 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.254168 15020 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:35.262413 15228 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.264074 14549 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.266562 14478 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.278143 14626 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
W20260806 20:34:35.281016 14595 consensus_peers.cc:487] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b370f85a8b944198972759bc1eb8061. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:35.281801 14369 catalog_manager.cc:5495] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.289779 14549 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.291059 14471 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:35.291157 15020 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:35.294354 15110 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:34:35.296303 14595 consensus_peers.cc:487] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 457a5ba876d14681a3fa2338fd9f6c4e. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:35.338456 15110 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.341176 14550 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.343103 14478 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.363518 14370 catalog_manager.cc:5495] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:35.442704 15246 ts_tablet_manager.cc:927] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:35.453728 15246 tablet_copy_client.cc:316] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:35.514151 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet f38960b1dd9245a8ba85fea601b09e94 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:35.514876 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet f38960b1dd9245a8ba85fea601b09e94 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-f38960b1dd9245a8ba85fea601b09e94
I20260806 20:34:35.523084 14489 tablet_copy_source_session.cc:205] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:35.526898 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38960b1dd9245a8ba85fea601b09e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:35.541743 15246 tablet_copy_client.cc:791] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:35.542443 15246 tablet_copy_client.cc:655] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:35.549654 15246 tablet_copy_client.cc:531] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:35.558851 15250 ts_tablet_manager.cc:927] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:35.563469 15250 tablet_copy_client.cc:316] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:35.565363 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:35.565781 15246 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:35.565929 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet a53ea52d2ecf44bd87a1c23ea5f5f8df from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-a53ea52d2ecf44bd87a1c23ea5f5f8df
I20260806 20:34:35.572345 14489 tablet_copy_source_session.cc:205] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:35.577858 15250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:35.591114 15246 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:35.592118 15246 tablet_bootstrap.cc:492] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:35.593755 15246 ts_tablet_manager.cc:1397] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.006s
I20260806 20:34:35.596549 15250 tablet_copy_client.cc:791] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:35.597297 15250 tablet_copy_client.cc:655] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:35.596827 15246 raft_consensus.cc:348] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.598692 15246 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:35.599396 15246 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.604195 15213 heartbeater.cc:503] Master 127.12.248.190:39439 was elected leader, sending a full tablet report...
I20260806 20:34:35.604287 15250 tablet_copy_client.cc:531] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:35.604518 15246 ts_tablet_manager.cc:1428] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 20:34:35.606241 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-f38960b1dd9245a8ba85fea601b09e94 received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:35.606638 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-f38960b1dd9245a8ba85fea601b09e94 on tablet f38960b1dd9245a8ba85fea601b09e94 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:35.616317 15250 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:35.643604 15250 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:35.644647 15250 tablet_bootstrap.cc:492] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:35.645367 15250 ts_tablet_manager.cc:1397] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.013s
I20260806 20:34:35.648124 15250 raft_consensus.cc:348] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.649017 15250 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:35.649694 15250 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.652335 15250 ts_tablet_manager.cc:1428] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 20:34:35.653956 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-a53ea52d2ecf44bd87a1c23ea5f5f8df received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:35.654346 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-a53ea52d2ecf44bd87a1c23ea5f5f8df on tablet a53ea52d2ecf44bd87a1c23ea5f5f8df with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:35.757915 15250 ts_tablet_manager.cc:927] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:35.759536 15250 tablet_copy_client.cc:316] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:35.781278 15246 ts_tablet_manager.cc:927] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:35.784123 15246 tablet_copy_client.cc:316] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:35.789619 14643 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet 6b370f85a8b944198972759bc1eb8061 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:35.789952 14642 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet 457a5ba876d14681a3fa2338fd9f6c4e from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:35.790176 14643 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet 6b370f85a8b944198972759bc1eb8061 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-6b370f85a8b944198972759bc1eb8061
I20260806 20:34:35.790876 14642 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet 457a5ba876d14681a3fa2338fd9f6c4e from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-457a5ba876d14681a3fa2338fd9f6c4e
I20260806 20:34:35.797770 14642 tablet_copy_source_session.cc:205] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:35.803900 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457a5ba876d14681a3fa2338fd9f6c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:35.821704 14643 tablet_copy_source_session.cc:205] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:35.825202 15250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b370f85a8b944198972759bc1eb8061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:35.832288 15246 tablet_copy_client.cc:791] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:35.833011 15246 tablet_copy_client.cc:655] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:35.837038 15246 tablet_copy_client.cc:531] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:35.851717 15250 tablet_copy_client.cc:791] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:35.853296 15246 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:35.853786 15250 tablet_copy_client.cc:655] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:35.859988 15250 tablet_copy_client.cc:531] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:35.875908 15187 raft_consensus.cc:1203] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:35.880398 15250 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:35.883939 15246 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:35.884958 15246 tablet_bootstrap.cc:492] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:35.885699 15246 ts_tablet_manager.cc:1397] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.010s
I20260806 20:34:35.890648 15246 raft_consensus.cc:348] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.891537 15246 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:35.892171 15246 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.894636 15246 ts_tablet_manager.cc:1428] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 20:34:35.907413 15250 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:35.908296 15250 tablet_bootstrap.cc:492] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:35.909294 15250 ts_tablet_manager.cc:1397] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.008s
I20260806 20:34:35.910588 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-457a5ba876d14681a3fa2338fd9f6c4e received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:35.911046 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-457a5ba876d14681a3fa2338fd9f6c4e on tablet 457a5ba876d14681a3fa2338fd9f6c4e with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:35.912649 15250 raft_consensus.cc:348] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.914045 15250 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:35.914932 15250 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:35.918195 15250 ts_tablet_manager.cc:1428] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260806 20:34:35.933372 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-6b370f85a8b944198972759bc1eb8061 received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:35.933810 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-6b370f85a8b944198972759bc1eb8061 on tablet 6b370f85a8b944198972759bc1eb8061 with peer 148bf2a11dba423ab364371f0458e871
W20260806 20:34:35.951841 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 45970.6us
I20260806 20:34:36.079699 15187 raft_consensus.cc:1203] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:36.170770 15187 raft_consensus.cc:1203] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:36.267289 15187 raft_consensus.cc:1203] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:36.337021 14924 raft_consensus.cc:1050] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:36.339354 14924 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.345683 15187 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.345870 14629 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.347222 15259 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:36.348403 14550 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.348388 14924 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:36.350355 15262 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:36.370482 15187 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.370683 14549 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.387625 14369 catalog_manager.cc:5495] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.368525 14924 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.408113 14627 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.440258 14476 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.445325 15187 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.446640 15259 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.449110 14629 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.450840 14924 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.457741 15262 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.461711 15187 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.475436 14627 raft_consensus.cc:2932] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.485308 14371 catalog_manager.cc:5495] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.488754 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f38960b1dd9245a8ba85fea601b09e94 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:36.502441 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet f38960b1dd9245a8ba85fea601b09e94 with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:36.507180 15271 tablet_replica.cc:330] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:36.508294 15271 raft_consensus.cc:2229] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:36.509027 15271 raft_consensus.cc:2258] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:36.513264 15271 ts_tablet_manager.cc:1905] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:36.529548 15271 ts_tablet_manager.cc:1918] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:36.530301 15271 log.cc:1196] T f38960b1dd9245a8ba85fea601b09e94 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/f38960b1dd9245a8ba85fea601b09e94
I20260806 20:34:36.531973 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet f38960b1dd9245a8ba85fea601b09e94 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:36.542553 15221 raft_consensus.cc:1050] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:36.544621 15221 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.554908 15187 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.555022 14626 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.556339 14551 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.557832 15259 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.558740 14924 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.560935 15221 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.575420 15221 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.575999 14550 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.586189 14627 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.599491 14372 catalog_manager.cc:5495] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:36.599403 15187 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.617241 14476 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.622176 15187 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.623286 14550 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.623731 15221 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.626658 15259 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.633766 14924 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.636543 15187 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.638278 15110 raft_consensus.cc:1050] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:36.639707 14550 raft_consensus.cc:2932] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.640281 15110 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.648952 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:36.654525 14370 catalog_manager.cc:5495] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.662456 14478 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.664391 15187 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.664269 15260 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:36.665747 15228 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.666594 14550 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.671587 15228 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.672957 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet a53ea52d2ecf44bd87a1c23ea5f5f8df with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:36.675639 15279 tablet_replica.cc:330] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:36.676677 15279 raft_consensus.cc:2229] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:36.679297 15279 raft_consensus.cc:2258] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:36.685803 15279 ts_tablet_manager.cc:1905] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:36.716487 15279 ts_tablet_manager.cc:1918] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:36.717010 15279 log.cc:1196] T a53ea52d2ecf44bd87a1c23ea5f5f8df P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/a53ea52d2ecf44bd87a1c23ea5f5f8df
I20260806 20:34:36.721040 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet a53ea52d2ecf44bd87a1c23ea5f5f8df (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:36.730428 15277 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.745765 14550 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.744917 14478 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.745899 15186 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.755437 14370 catalog_manager.cc:5495] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:36.779142 14633 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.786109 15186 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.786660 14479 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.789426 15228 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.792599 15110 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.800580 15110 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.803359 14479 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.803373 15186 raft_consensus.cc:2932] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.813706 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 457a5ba876d14681a3fa2338fd9f6c4e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:36.818334 14372 catalog_manager.cc:5495] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.842947 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 457a5ba876d14681a3fa2338fd9f6c4e with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:36.843892 15271 tablet_replica.cc:330] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:36.844988 15271 raft_consensus.cc:2229] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:36.845695 15271 raft_consensus.cc:2258] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:36.852229 15271 ts_tablet_manager.cc:1905] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:36.865510 15277 raft_consensus.cc:1050] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:36.868373 15271 ts_tablet_manager.cc:1918] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:36.867977 15277 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.868861 15271 log.cc:1196] T 457a5ba876d14681a3fa2338fd9f6c4e P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/457a5ba876d14681a3fa2338fd9f6c4e
I20260806 20:34:36.870879 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet 457a5ba876d14681a3fa2338fd9f6c4e (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:36.875857 14549 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.876607 15186 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.877051 14478 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:36.877782 15277 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.878696 15110 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.880164 15228 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:36.895959 15283 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.897217 14549 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.899868 14478 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.909795 15186 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.914746 14371 catalog_manager.cc:5495] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:36.947412 14629 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:36.954329 14550 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.955467 15186 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:36.956600 15283 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.957511 15110 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:36.963848 15228 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.970939 15186 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.979182 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6b370f85a8b944198972759bc1eb8061 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:36.983515 14371 catalog_manager.cc:5495] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:36.983553 14550 raft_consensus.cc:2932] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:36.993228 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b370f85a8b944198972759bc1eb8061 with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
I20260806 20:34:36.995687 15287 tablet_replica.cc:330] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:36.996682 15287 raft_consensus.cc:2229] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:36.997660 15287 raft_consensus.cc:2258] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:37.001155 15287 ts_tablet_manager.cc:1905] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:37.016891 15287 ts_tablet_manager.cc:1918] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:37.017249 15287 log.cc:1196] T 6b370f85a8b944198972759bc1eb8061 P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/6b370f85a8b944198972759bc1eb8061
I20260806 20:34:37.019188 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet 6b370f85a8b944198972759bc1eb8061 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:38.056056 14627 consensus_queue.cc:237] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.061434 14550 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:38.062001 14595 consensus_peers.cc:487] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a9d79d20363456d80521feddfec49bc. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:38.062261 14478 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:38.062671 15277 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:38.064386 15277 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:38.073879 15277 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.076896 14550 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.081735 14478 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.094190 14370 catalog_manager.cc:5495] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.102885 14478 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.110738 14627 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:38.111518 14441 consensus_peers.cc:487] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3694b5a404a94fbea69b831722e2d3e4. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:38.112133 15262 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:38.115271 14550 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:38.116532 15262 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:38.135784 15259 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.137426 14629 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.139477 14550 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.153683 14627 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.162938 14479 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:38.163900 14550 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:38.164201 14595 consensus_peers.cc:487] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87adab531acb4032af0c9a18dd9893e0. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:38.165354 15293 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:38.154260 14369 catalog_manager.cc:5495] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:38.168123 15293 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:38.186452 15277 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.188868 14479 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.195693 14549 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.206976 14369 catalog_manager.cc:5495] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:38.229996 14549 consensus_queue.cc:237] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.241516 14478 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:38.246023 14627 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:38.248538 15249 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:38.249869 15288 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:34:38.263700 14518 consensus_peers.cc:487] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b84df1548a8f46c492d5379377b5bc29. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:38.265524 15288 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.267130 14476 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.278322 14629 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.297109 14369 catalog_manager.cc:5495] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:38.553771 15311 ts_tablet_manager.cc:927] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:38.555740 15311 tablet_copy_client.cc:316] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:38.557197 14643 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet 87adab531acb4032af0c9a18dd9893e0 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:38.557718 14643 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet 87adab531acb4032af0c9a18dd9893e0 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-87adab531acb4032af0c9a18dd9893e0
I20260806 20:34:38.564716 14643 tablet_copy_source_session.cc:205] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:38.567466 15311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87adab531acb4032af0c9a18dd9893e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:38.580476 15311 tablet_copy_client.cc:791] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:38.581034 15311 tablet_copy_client.cc:655] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:38.585780 15311 tablet_copy_client.cc:531] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:38.594220 15311 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:38.613456 15311 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:38.614336 15311 tablet_bootstrap.cc:492] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:38.614970 15311 ts_tablet_manager.cc:1397] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20260806 20:34:38.632128 15311 raft_consensus.cc:348] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.633139 15311 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:38.633885 15311 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.650668 15311 ts_tablet_manager.cc:1428] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.035s user 0.018s sys 0.009s
I20260806 20:34:38.653328 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-87adab531acb4032af0c9a18dd9893e0 received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:38.653728 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-87adab531acb4032af0c9a18dd9893e0 on tablet 87adab531acb4032af0c9a18dd9893e0 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:38.656200 15311 ts_tablet_manager.cc:927] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:38.656824 15315 ts_tablet_manager.cc:927] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:38.658170 15311 tablet_copy_client.cc:316] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:38.660564 14643 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet 6a9d79d20363456d80521feddfec49bc from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:38.661077 14643 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet 6a9d79d20363456d80521feddfec49bc from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-6a9d79d20363456d80521feddfec49bc
I20260806 20:34:38.667311 14643 tablet_copy_source_session.cc:205] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:38.671465 15311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d79d20363456d80521feddfec49bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:38.673745 15315 tablet_copy_client.cc:316] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:38.675287 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet 3694b5a404a94fbea69b831722e2d3e4 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:38.675799 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet 3694b5a404a94fbea69b831722e2d3e4 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-3694b5a404a94fbea69b831722e2d3e4
I20260806 20:34:38.683513 14489 tablet_copy_source_session.cc:205] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:38.686856 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694b5a404a94fbea69b831722e2d3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:38.691479 15311 tablet_copy_client.cc:791] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:38.693153 15311 tablet_copy_client.cc:655] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:38.699239 15311 tablet_copy_client.cc:531] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:38.704480 15315 tablet_copy_client.cc:791] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:38.708947 15311 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:38.705078 15315 tablet_copy_client.cc:655] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:38.717309 15315 tablet_copy_client.cc:531] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:38.730471 15315 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:38.736477 15311 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:38.737402 15311 tablet_bootstrap.cc:492] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:38.738054 15311 ts_tablet_manager.cc:1397] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.001s
I20260806 20:34:38.740873 15311 raft_consensus.cc:348] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.741766 15311 raft_consensus.cc:729] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:38.742440 15311 consensus_queue.cc:260] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.745293 15311 ts_tablet_manager.cc:1428] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 20:34:38.749960 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-6a9d79d20363456d80521feddfec49bc received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:38.750367 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-6a9d79d20363456d80521feddfec49bc on tablet 6a9d79d20363456d80521feddfec49bc with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:38.755707 15315 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:38.756704 15315 tablet_bootstrap.cc:492] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:38.757391 15315 ts_tablet_manager.cc:1397] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.005s
I20260806 20:34:38.759991 15315 raft_consensus.cc:348] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.760787 15315 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:38.761734 15311 ts_tablet_manager.cc:927] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:38.761440 15315 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.764220 15315 ts_tablet_manager.cc:1428] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 20:34:38.765944 15311 tablet_copy_client.cc:316] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:38.766053 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-3694b5a404a94fbea69b831722e2d3e4 received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:38.766537 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-3694b5a404a94fbea69b831722e2d3e4 on tablet 3694b5a404a94fbea69b831722e2d3e4 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:38.795735 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet b84df1548a8f46c492d5379377b5bc29 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:38.796360 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet b84df1548a8f46c492d5379377b5bc29 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-b84df1548a8f46c492d5379377b5bc29
I20260806 20:34:38.804399 14567 tablet_copy_source_session.cc:205] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:38.807516 15311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84df1548a8f46c492d5379377b5bc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:38.822055 15311 tablet_copy_client.cc:791] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:38.822723 15311 tablet_copy_client.cc:655] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:38.827773 15311 tablet_copy_client.cc:531] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:38.839088 15311 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:38.856791 15311 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:38.857475 15311 tablet_bootstrap.cc:492] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:38.858099 15311 ts_tablet_manager.cc:1397] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.001s
I20260806 20:34:38.860388 15311 raft_consensus.cc:348] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.861181 15311 raft_consensus.cc:729] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:38.861792 15311 consensus_queue.cc:260] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:38.863849 15311 ts_tablet_manager.cc:1428] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 20:34:38.865832 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-b84df1548a8f46c492d5379377b5bc29 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:38.866179 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-b84df1548a8f46c492d5379377b5bc29 on tablet b84df1548a8f46c492d5379377b5bc29 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:38.988831 15187 raft_consensus.cc:1203] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:39.212435 15187 raft_consensus.cc:1203] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:39.249562 15187 raft_consensus.cc:1203] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:39.301425 15187 raft_consensus.cc:1203] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:39.415850 15293 raft_consensus.cc:1050] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:39.418360 15293 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:39.426205 14549 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.426376 15187 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.427474 15228 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.428586 15277 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.432729 14478 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.435160 15277 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.437135 15293 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.441406 14550 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.456393 14369 catalog_manager.cc:5495] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.456961 15187 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.469375 14478 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.507628 14627 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:39.513821 15187 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:39.513880 14478 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:39.515094 15293 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:39.515969 15329 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:39.521956 15277 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.523442 14479 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.526080 15187 raft_consensus.cc:2932] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.537997 14370 catalog_manager.cc:5495] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.548892 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 87adab531acb4032af0c9a18dd9893e0 with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:39.550305 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 87adab531acb4032af0c9a18dd9893e0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:39.552246 15337 tablet_replica.cc:330] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:39.553377 15337 raft_consensus.cc:2229] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:39.554111 15337 raft_consensus.cc:2258] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:39.557020 15337 ts_tablet_manager.cc:1905] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:39.574007 15337 ts_tablet_manager.cc:1918] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:39.574394 15337 log.cc:1196] T 87adab531acb4032af0c9a18dd9893e0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/87adab531acb4032af0c9a18dd9893e0
I20260806 20:34:39.575810 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet 87adab531acb4032af0c9a18dd9893e0 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:39.627164 15289 raft_consensus.cc:1050] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:39.629459 15289 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:39.635905 15187 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.635905 14627 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.636824 14549 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.637527 15289 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.638368 15327 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.639293 15259 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.650239 15289 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.652305 14627 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.655203 15187 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.670207 14549 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.675982 14370 catalog_manager.cc:5495] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:39.703233 14478 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:39.707912 15187 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:39.709354 15289 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:39.710999 14549 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:39.712718 15259 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:39.715943 15327 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.723176 14549 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.728508 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3694b5a404a94fbea69b831722e2d3e4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:39.735008 15187 raft_consensus.cc:2932] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.741098 14370 catalog_manager.cc:5495] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.754282 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet 3694b5a404a94fbea69b831722e2d3e4 with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:39.773293 15343 tablet_replica.cc:330] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:39.774214 15343 raft_consensus.cc:2229] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:39.775120 15343 raft_consensus.cc:2258] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:39.778636 15343 ts_tablet_manager.cc:1905] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:39.793449 15343 ts_tablet_manager.cc:1918] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:39.793835 15343 log.cc:1196] T 3694b5a404a94fbea69b831722e2d3e4 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/3694b5a404a94fbea69b831722e2d3e4
I20260806 20:34:39.794723 15277 raft_consensus.cc:1050] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:39.797015 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet 3694b5a404a94fbea69b831722e2d3e4 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:39.797142 15277 consensus_queue.cc:237] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:39.807449 15187 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.807893 14549 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.808686 14478 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.808667 15277 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:39.809705 15293 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.810559 15228 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.819927 15228 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.821695 14551 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.825798 15186 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.841207 14372 catalog_manager.cc:5495] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.866762 14478 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.889062 14633 consensus_queue.cc:237] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:39.895246 15187 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:39.896381 15293 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:39.897451 14550 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:39.899819 15293 consensus_queue.cc:1035] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:39.903172 15288 raft_consensus.cc:1050] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:39.906085 15288 consensus_queue.cc:237] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:39.909158 15228 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.915988 14550 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.926997 15187 raft_consensus.cc:2932] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.947248 14372 catalog_manager.cc:5495] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:39.947959 14478 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.948988 15187 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.949501 15352 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.950522 15288 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:39.959887 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9d79d20363456d80521feddfec49bc with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
I20260806 20:34:39.961840 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6a9d79d20363456d80521feddfec49bc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260806 20:34:39.965466 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 46198.2us
I20260806 20:34:39.967160 14633 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:39.982115 15354 tablet_replica.cc:330] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:39.983155 15354 raft_consensus.cc:2229] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:39.992506 15354 raft_consensus.cc:2258] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:40.002971 15288 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.004037 14476 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.015607 15186 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.023516 15354 ts_tablet_manager.cc:1905] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:40.029814 14372 catalog_manager.cc:5495] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.039743 15288 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:40.054505 14633 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.065896 15354 ts_tablet_manager.cc:1918] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:40.066378 15354 log.cc:1196] T 6a9d79d20363456d80521feddfec49bc P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/6a9d79d20363456d80521feddfec49bc
I20260806 20:34:40.067785 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet 6a9d79d20363456d80521feddfec49bc (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:40.484942 14552 consensus_queue.cc:237] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:40.489162 15186 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:40.490633 15351 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:40.491019 14627 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:40.492347 15356 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:40.497143 15351 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.498696 15187 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.507215 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b84df1548a8f46c492d5379377b5bc29 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:40.507947 14627 raft_consensus.cc:2932] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.510231 14369 catalog_manager.cc:5495] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:40.527671 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet b84df1548a8f46c492d5379377b5bc29 with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
I20260806 20:34:40.528664 15354 tablet_replica.cc:330] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:40.529700 15354 raft_consensus.cc:2229] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:40.530308 15354 raft_consensus.cc:2258] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:40.533748 15354 ts_tablet_manager.cc:1905] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:40.548372 15354 ts_tablet_manager.cc:1918] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:40.548933 15354 log.cc:1196] T b84df1548a8f46c492d5379377b5bc29 P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/b84df1548a8f46c492d5379377b5bc29
I20260806 20:34:40.550333 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet b84df1548a8f46c492d5379377b5bc29 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:41.583204 14552 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:41.588696 14627 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:41.589309 14518 consensus_peers.cc:487] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d7dc7fd91be46ecb624803a9b029e5b. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:41.590277 14478 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:41.590060 15288 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:41.591396 15360 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:41.598135 15351 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.600050 14627 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.600598 14478 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.611337 14479 consensus_queue.cc:237] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:41.617138 14369 catalog_manager.cc:5495] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:34:41.624114 14441 consensus_peers.cc:487] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d01e9209cd924024ae8e7e630b5accd3. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:41.624588 14633 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:41.625903 15289 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:41.626470 14552 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:41.627933 15289 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:41.637732 15289 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.642390 14627 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.664855 14629 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:41.681746 14552 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.720223 14551 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:41.721647 15348 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:41.722985 14479 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:41.728566 15228 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:41.715220 14370 catalog_manager.cc:5495] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:34:41.738232 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 72533.7us
I20260806 20:34:41.748409 15348 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.750743 14550 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
W20260806 20:34:41.761016 14595 consensus_peers.cc:487] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17eec21790894c559ad9623765626416. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:41.770700 14629 consensus_queue.cc:237] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:41.788638 14478 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.789129 14372 catalog_manager.cc:5495] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.793937 14550 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:41.796331 15376 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:41.801739 14479 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:41.802996 14595 consensus_peers.cc:487] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db55591983ce42868e62df33ee1b56be. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:41.803853 15376 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 20:34:41.809408 15228 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.812414 14552 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.827739 14479 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:41.828079 14372 catalog_manager.cc:5495] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:42.106396 15382 ts_tablet_manager.cc:927] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:42.108543 15382 tablet_copy_client.cc:316] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:42.110069 14643 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet 17eec21790894c559ad9623765626416 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:42.110611 14643 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet 17eec21790894c559ad9623765626416 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-17eec21790894c559ad9623765626416
I20260806 20:34:42.123021 14643 tablet_copy_source_session.cc:205] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:42.126298 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17eec21790894c559ad9623765626416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:42.141382 15384 ts_tablet_manager.cc:927] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:42.146834 15382 tablet_copy_client.cc:791] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:42.147570 15382 tablet_copy_client.cc:655] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:42.149955 15384 tablet_copy_client.cc:316] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:42.152374 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 6d7dc7fd91be46ecb624803a9b029e5b from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:42.152596 15382 tablet_copy_client.cc:531] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:42.152940 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 6d7dc7fd91be46ecb624803a9b029e5b from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-6d7dc7fd91be46ecb624803a9b029e5b
I20260806 20:34:42.160364 14567 tablet_copy_source_session.cc:205] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:42.161868 15382 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:42.164019 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7dc7fd91be46ecb624803a9b029e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:42.177852 15384 tablet_copy_client.cc:791] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:42.178403 15384 tablet_copy_client.cc:655] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:42.186038 15384 tablet_copy_client.cc:531] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:42.187500 15382 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:42.188423 15382 tablet_bootstrap.cc:492] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:42.189105 15382 ts_tablet_manager.cc:1397] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.006s
I20260806 20:34:42.191615 15382 raft_consensus.cc:348] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.192442 15382 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:42.193080 15382 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.194995 15384 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:42.206287 15382 ts_tablet_manager.cc:1428] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260806 20:34:42.208751 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-17eec21790894c559ad9623765626416 received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:42.209148 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-17eec21790894c559ad9623765626416 on tablet 17eec21790894c559ad9623765626416 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:42.219693 15384 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:42.220603 15384 tablet_bootstrap.cc:492] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:42.221269 15384 ts_tablet_manager.cc:1397] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.004s
I20260806 20:34:42.224121 15384 raft_consensus.cc:348] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.224978 15384 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:42.225589 15384 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.227861 15384 ts_tablet_manager.cc:1428] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:34:42.233623 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-6d7dc7fd91be46ecb624803a9b029e5b received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:42.234041 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-6d7dc7fd91be46ecb624803a9b029e5b on tablet 6d7dc7fd91be46ecb624803a9b029e5b with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:42.245368 15384 ts_tablet_manager.cc:927] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:42.256673 15384 tablet_copy_client.cc:316] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:42.258203 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet d01e9209cd924024ae8e7e630b5accd3 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:42.258734 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet d01e9209cd924024ae8e7e630b5accd3 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-d01e9209cd924024ae8e7e630b5accd3
I20260806 20:34:42.266265 14489 tablet_copy_source_session.cc:205] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:42.270959 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01e9209cd924024ae8e7e630b5accd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:42.285164 15384 tablet_copy_client.cc:791] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:42.285668 15384 tablet_copy_client.cc:655] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:42.289198 15384 tablet_copy_client.cc:531] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:42.297339 15384 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:42.317344 15384 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:42.318228 15384 tablet_bootstrap.cc:492] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:42.318919 15384 ts_tablet_manager.cc:1397] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.012s
I20260806 20:34:42.321642 15384 raft_consensus.cc:348] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.322340 15384 raft_consensus.cc:729] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:42.323004 15384 consensus_queue.cc:260] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.325153 15384 ts_tablet_manager.cc:1428] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 20:34:42.327297 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-d01e9209cd924024ae8e7e630b5accd3 received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:42.327690 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-d01e9209cd924024ae8e7e630b5accd3 on tablet d01e9209cd924024ae8e7e630b5accd3 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:42.416198 15384 ts_tablet_manager.cc:927] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:42.417902 15384 tablet_copy_client.cc:316] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:42.419430 14643 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet db55591983ce42868e62df33ee1b56be from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:42.420133 14643 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet db55591983ce42868e62df33ee1b56be from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-db55591983ce42868e62df33ee1b56be
I20260806 20:34:42.430574 14643 tablet_copy_source_session.cc:205] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:42.434171 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55591983ce42868e62df33ee1b56be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:42.451107 15384 tablet_copy_client.cc:791] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:42.451710 15384 tablet_copy_client.cc:655] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:42.523054 15384 tablet_copy_client.cc:531] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:42.544904 15187 raft_consensus.cc:1203] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260806 20:34:42.569132 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48158.3us
I20260806 20:34:42.574613 15384 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:42.600719 15384 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:42.603420 15384 tablet_bootstrap.cc:492] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:42.604207 15384 ts_tablet_manager.cc:1397] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20260806 20:34:42.609117 15384 raft_consensus.cc:348] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.610777 15384 raft_consensus.cc:729] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:42.611547 15384 consensus_queue.cc:260] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:42.618076 15384 ts_tablet_manager.cc:1428] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260806 20:34:42.620847 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-db55591983ce42868e62df33ee1b56be received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:42.621379 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-db55591983ce42868e62df33ee1b56be on tablet db55591983ce42868e62df33ee1b56be with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:42.667397 15187 raft_consensus.cc:1203] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:42.708281 15187 raft_consensus.cc:1203] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:42.874820 15187 raft_consensus.cc:1203] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:43.018707 15398 raft_consensus.cc:1050] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:43.020969 15398 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.030841 15187 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.031924 14479 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.029803 14552 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.034260 15348 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:43.035910 15348 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:43.037647 15397 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:43.062201 15398 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.066946 15187 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.068132 14552 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.081307 14369 catalog_manager.cc:5495] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.091310 15391 raft_consensus.cc:1050] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:43.094081 15391 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.091794 14479 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.097286 15289 raft_consensus.cc:1050] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:43.099802 15289 consensus_queue.cc:237] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.137030 15187 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.144376 15342 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:43.149328 14552 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.149497 14633 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.154774 15186 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.156172 15391 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:34:43.157922 14479 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.158844 14633 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.159209 15342 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:43.161993 15372 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:43.162142 15374 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:43.163851 15374 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:43.263983 15374 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.268637 15398 raft_consensus.cc:1050] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:43.273311 15398 consensus_queue.cc:237] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.286767 14478 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.309934 15342 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.313779 15187 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.320007 14627 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.340910 15186 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.342437 14552 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.344633 15398 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:34:43.346927 14471 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:43.346930 15397 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:43.348681 15348 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 20:34:43.359509 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 101549us
I20260806 20:34:43.379792 15187 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.379635 14625 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.398550 14552 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.423192 15397 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.430485 15186 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.435348 14471 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.440210 14551 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.461870 14371 catalog_manager.cc:5495] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.470093 14625 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.471517 14371 catalog_manager.cc:5495] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.473294 14371 catalog_manager.cc:5495] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.491607 14552 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.493299 15186 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.494972 15397 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:43.496042 15396 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:43.501103 14550 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.514437 15187 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.515033 15396 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.516647 15420 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:43.519675 14471 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.520630 14478 consensus_queue.cc:237] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.524796 14633 consensus_queue.cc:237] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:43.529086 14550 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.530920 15186 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.531797 14551 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.534376 15342 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:43.534597 15391 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:43.540194 15342 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:43.539693 15185 raft_consensus.cc:2932] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.546073 15372 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.553779 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 17eec21790894c559ad9623765626416 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:43.555311 15187 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.555578 15415 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.561555 15184 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.565956 14478 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.568341 15396 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:43.575814 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d01e9209cd924024ae8e7e630b5accd3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:43.576630 15398 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.578523 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6d7dc7fd91be46ecb624803a9b029e5b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:43.582021 15184 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.590236 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet db55591983ce42868e62df33ee1b56be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:43.588927 14471 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:43.593046 15398 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:43.593361 14550 raft_consensus.cc:2932] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.580497 14371 catalog_manager.cc:5495] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.601933 14476 raft_consensus.cc:2932] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.621800 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 17eec21790894c559ad9623765626416 with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
I20260806 20:34:43.620226 15184 raft_consensus.cc:2932] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.634675 14371 catalog_manager.cc:5495] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.636445 14371 catalog_manager.cc:5495] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:43.639443 15425 tablet_replica.cc:330] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:43.640614 15425 raft_consensus.cc:2229] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:43.641382 15425 raft_consensus.cc:2258] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:43.649837 15425 ts_tablet_manager.cc:1905] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:43.653607 14370 catalog_manager.cc:5495] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:43.661139 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet d01e9209cd924024ae8e7e630b5accd3 with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:43.662490 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet db55591983ce42868e62df33ee1b56be with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:43.668170 15426 tablet_replica.cc:330] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:43.671795 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7dc7fd91be46ecb624803a9b029e5b with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:43.673815 15427 tablet_replica.cc:330] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:43.675518 15427 raft_consensus.cc:2229] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:43.669621 15426 raft_consensus.cc:2229] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:43.680779 15426 raft_consensus.cc:2258] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:43.684523 15425 ts_tablet_manager.cc:1918] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:43.685113 15425 log.cc:1196] T 17eec21790894c559ad9623765626416 P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/17eec21790894c559ad9623765626416
I20260806 20:34:43.686882 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet 17eec21790894c559ad9623765626416 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:43.688584 15427 raft_consensus.cc:2258] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:43.692830 15427 ts_tablet_manager.cc:1905] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:43.696661 15426 ts_tablet_manager.cc:1905] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:43.712085 15427 ts_tablet_manager.cc:1918] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:43.712729 15427 log.cc:1196] T db55591983ce42868e62df33ee1b56be P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/db55591983ce42868e62df33ee1b56be
I20260806 20:34:43.711475 15426 ts_tablet_manager.cc:1918] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:43.714430 15426 log.cc:1196] T d01e9209cd924024ae8e7e630b5accd3 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/d01e9209cd924024ae8e7e630b5accd3
I20260806 20:34:43.714879 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet db55591983ce42868e62df33ee1b56be (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:43.716257 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet d01e9209cd924024ae8e7e630b5accd3 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:43.716442 15426 tablet_replica.cc:330] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:43.717455 15426 raft_consensus.cc:2229] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:43.718053 15426 raft_consensus.cc:2258] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:43.720631 15426 ts_tablet_manager.cc:1905] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:43.732650 15426 ts_tablet_manager.cc:1918] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:43.732965 15426 log.cc:1196] T 6d7dc7fd91be46ecb624803a9b029e5b P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/6d7dc7fd91be46ecb624803a9b029e5b
I20260806 20:34:43.734157 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet 6d7dc7fd91be46ecb624803a9b029e5b (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:44.725841 14478 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:44.731284 14625 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:44.732012 14441 consensus_peers.cc:487] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7815d5557ccf4e569039960d0a8f24ba. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:44.732797 15423 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.734092 14551 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:44.735244 15423 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.742514 15429 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.743984 14625 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.745347 14551 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.763300 14370 catalog_manager.cc:5495] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.781883 14551 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:44.796607 14625 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:44.797221 14478 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:44.798449 14518 consensus_peers.cc:487] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8933c210fdc40cab4369e6aa56f7020. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:44.798636 15415 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.799554 15420 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.809087 15415 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.810709 14625 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.812315 14478 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.822697 14551 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:44.827787 14369 catalog_manager.cc:5495] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.837405 14625 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:44.839514 14518 consensus_peers.cc:487] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8a0e4079ebe47aaa3c275ed7f559ab8. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:44.840191 14478 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:44.841485 15415 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.844295 15437 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.861164 15420 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.864188 14476 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.870304 14625 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.880010 14371 catalog_manager.cc:5495] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:44.882135 14478 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
W20260806 20:34:44.895854 14441 consensus_peers.cc:487] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc270596f078400e9e1b181a9706b516. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:44.896441 14625 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:44.897835 15429 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.901844 14550 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:44.903680 15429 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:44.908738 15429 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.910202 14633 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.923496 14550 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:44.929648 14370 catalog_manager.cc:5495] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:45.317796 15446 ts_tablet_manager.cc:927] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:45.319880 15446 tablet_copy_client.cc:316] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:45.321972 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet d8933c210fdc40cab4369e6aa56f7020 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:45.322492 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet d8933c210fdc40cab4369e6aa56f7020 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-d8933c210fdc40cab4369e6aa56f7020
I20260806 20:34:45.331298 14567 tablet_copy_source_session.cc:205] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:45.331880 15448 ts_tablet_manager.cc:927] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:45.334141 15448 tablet_copy_client.cc:316] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:45.335804 15446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8933c210fdc40cab4369e6aa56f7020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:45.336004 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet 7815d5557ccf4e569039960d0a8f24ba from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:45.336614 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet 7815d5557ccf4e569039960d0a8f24ba from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-7815d5557ccf4e569039960d0a8f24ba
I20260806 20:34:45.344408 14489 tablet_copy_source_session.cc:205] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:45.349541 15448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7815d5557ccf4e569039960d0a8f24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:45.351267 15446 tablet_copy_client.cc:791] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:45.351903 15446 tablet_copy_client.cc:655] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:45.355502 15446 tablet_copy_client.cc:531] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:45.362629 15446 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:45.366576 15448 tablet_copy_client.cc:791] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:45.367283 15448 tablet_copy_client.cc:655] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:45.371366 15448 tablet_copy_client.cc:531] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:45.382615 15446 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:45.382615 15448 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:45.383685 15446 tablet_bootstrap.cc:492] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:45.384505 15446 ts_tablet_manager.cc:1397] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.015s
I20260806 20:34:45.387159 15446 raft_consensus.cc:348] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.388123 15446 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:45.388842 15446 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.398870 15446 ts_tablet_manager.cc:1428] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260806 20:34:45.401074 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-d8933c210fdc40cab4369e6aa56f7020 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:45.401469 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-d8933c210fdc40cab4369e6aa56f7020 on tablet d8933c210fdc40cab4369e6aa56f7020 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:45.414739 15448 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:45.415700 15448 tablet_bootstrap.cc:492] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:45.416392 15448 ts_tablet_manager.cc:1397] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.004s
I20260806 20:34:45.419059 15448 raft_consensus.cc:348] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.420107 15448 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:45.420827 15448 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.423357 15448 ts_tablet_manager.cc:1428] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:34:45.425029 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-7815d5557ccf4e569039960d0a8f24ba received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:45.425379 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-7815d5557ccf4e569039960d0a8f24ba on tablet 7815d5557ccf4e569039960d0a8f24ba with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:45.437083 15448 ts_tablet_manager.cc:927] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:45.438938 15448 tablet_copy_client.cc:316] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:45.440745 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet d8a0e4079ebe47aaa3c275ed7f559ab8 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:45.441273 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet d8a0e4079ebe47aaa3c275ed7f559ab8 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-d8a0e4079ebe47aaa3c275ed7f559ab8
I20260806 20:34:45.447956 14567 tablet_copy_source_session.cc:205] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:45.450593 15448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a0e4079ebe47aaa3c275ed7f559ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:45.470561 15446 ts_tablet_manager.cc:927] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:45.471052 15448 tablet_copy_client.cc:791] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:45.471676 15448 tablet_copy_client.cc:655] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:45.475649 15448 tablet_copy_client.cc:531] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:45.485424 15448 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:45.487921 15446 tablet_copy_client.cc:316] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:45.490298 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet dc270596f078400e9e1b181a9706b516 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:45.490798 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet dc270596f078400e9e1b181a9706b516 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-dc270596f078400e9e1b181a9706b516
I20260806 20:34:45.503103 14489 tablet_copy_source_session.cc:205] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:45.509583 15446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc270596f078400e9e1b181a9706b516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:45.511741 15448 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:45.512624 15448 tablet_bootstrap.cc:492] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:45.513281 15448 ts_tablet_manager.cc:1397] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.000s
I20260806 20:34:45.516435 15448 raft_consensus.cc:348] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.517299 15448 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:45.517907 15448 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.520354 15448 ts_tablet_manager.cc:1428] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 20:34:45.522851 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-d8a0e4079ebe47aaa3c275ed7f559ab8 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:45.523239 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-d8a0e4079ebe47aaa3c275ed7f559ab8 on tablet d8a0e4079ebe47aaa3c275ed7f559ab8 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:45.529201 15446 tablet_copy_client.cc:791] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:45.529827 15446 tablet_copy_client.cc:655] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:45.533314 15446 tablet_copy_client.cc:531] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:45.540982 15446 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:45.560097 15446 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:45.560873 15446 tablet_bootstrap.cc:492] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:45.561579 15446 ts_tablet_manager.cc:1397] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
I20260806 20:34:45.564157 15446 raft_consensus.cc:348] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.564947 15446 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:45.565631 15446 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:45.568482 15446 ts_tablet_manager.cc:1428] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:34:45.573988 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-dc270596f078400e9e1b181a9706b516 received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:45.574406 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-dc270596f078400e9e1b181a9706b516 on tablet dc270596f078400e9e1b181a9706b516 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:45.838490 15186 raft_consensus.cc:1203] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:45.890440 15186 raft_consensus.cc:1203] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:45.919143 15186 raft_consensus.cc:1203] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:45.985230 15186 raft_consensus.cc:1203] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:46.368589 15423 raft_consensus.cc:1050] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:46.371016 15423 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.380498 15186 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.381227 14633 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.382018 15423 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:46.383170 15429 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.384409 15460 raft_consensus.cc:1050] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:46.386756 15460 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.393255 14551 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.395079 15423 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:46.401183 14626 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.403370 15459 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.413429 15187 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.413935 14471 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.415575 15459 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.419616 15460 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:46.431038 15429 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.437999 14633 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.439163 14552 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.442421 15184 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.451318 14369 catalog_manager.cc:5495] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:46.452242 15437 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.476752 14626 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.488269 15429 raft_consensus.cc:1050] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:46.483214 14471 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.494050 15184 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.490841 15429 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.518474 14626 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.519254 15186 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.520231 15423 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:46.521420 15475 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.539395 14552 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.542400 15423 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.546742 15471 raft_consensus.cc:1050] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:46.549475 15471 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.555274 15429 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.559247 14633 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.559618 14552 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.560611 14471 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.561771 15415 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.565078 15186 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:46.566517 15415 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.568653 15470 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:46.579416 15186 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.585554 14476 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.585228 14626 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.599658 14372 catalog_manager.cc:5495] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.609467 15184 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.610324 14633 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.611899 15457 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.625458 15423 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.627120 15184 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.629524 15429 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.636552 14625 raft_consensus.cc:2932] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.639036 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7815d5557ccf4e569039960d0a8f24ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:46.640604 14370 catalog_manager.cc:5495] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.648407 15470 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.652243 14476 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.664556 14626 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.666105 14369 catalog_manager.cc:5495] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:46.670536 14550 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.680742 15184 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.681561 14471 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.684206 15471 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.685855 15437 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.697068 15415 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.701848 15184 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.704713 14476 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.705185 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 7815d5557ccf4e569039960d0a8f24ba with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:46.709851 14478 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.713676 15186 raft_consensus.cc:2932] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.718928 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d8933c210fdc40cab4369e6aa56f7020 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:46.724068 15187 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.725667 15457 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.729084 14626 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.730758 15457 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.710376 14371 catalog_manager.cc:5495] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:46.742887 15457 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.744819 14626 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.748203 15485 tablet_replica.cc:330] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:46.749536 15485 raft_consensus.cc:2229] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:46.750214 15485 raft_consensus.cc:2258] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:46.758456 15485 ts_tablet_manager.cc:1905] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:46.770850 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dc270596f078400e9e1b181a9706b516 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:46.774247 14371 catalog_manager.cc:5495] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.781548 15485 ts_tablet_manager.cc:1918] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:46.782056 15485 log.cc:1196] T 7815d5557ccf4e569039960d0a8f24ba P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/7815d5557ccf4e569039960d0a8f24ba
I20260806 20:34:46.783905 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet 7815d5557ccf4e569039960d0a8f24ba (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:46.754554 15186 raft_consensus.cc:2932] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.803999 14370 catalog_manager.cc:5495] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:46.814109 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet d8933c210fdc40cab4369e6aa56f7020 with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:46.822436 15486 tablet_replica.cc:330] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:46.828820 15486 raft_consensus.cc:2229] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:46.830018 15486 raft_consensus.cc:2258] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:46.836010 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet dc270596f078400e9e1b181a9706b516 with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:46.836483 14550 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:46.837765 15485 tablet_replica.cc:330] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:46.848503 15486 ts_tablet_manager.cc:1905] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:46.854094 15186 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.855605 14626 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:46.856544 15415 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.858680 15470 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:46.867235 15486 ts_tablet_manager.cc:1918] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:46.867784 15486 log.cc:1196] T d8933c210fdc40cab4369e6aa56f7020 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/d8933c210fdc40cab4369e6aa56f7020
I20260806 20:34:46.869663 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet d8933c210fdc40cab4369e6aa56f7020 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:46.868897 15437 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.872969 15485 raft_consensus.cc:2229] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:46.873708 15485 raft_consensus.cc:2258] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:46.874302 15186 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.873667 14626 raft_consensus.cc:2932] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.885232 15485 ts_tablet_manager.cc:1905] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:46.892004 14368 catalog_manager.cc:5495] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:46.896067 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d8a0e4079ebe47aaa3c275ed7f559ab8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:46.911671 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a0e4079ebe47aaa3c275ed7f559ab8 with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
I20260806 20:34:46.929189 15487 tablet_replica.cc:330] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:46.930294 15485 ts_tablet_manager.cc:1918] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:46.930467 15487 raft_consensus.cc:2229] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:46.930871 15485 log.cc:1196] T dc270596f078400e9e1b181a9706b516 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/dc270596f078400e9e1b181a9706b516
I20260806 20:34:46.931473 15487 raft_consensus.cc:2258] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:46.933638 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet dc270596f078400e9e1b181a9706b516 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:46.935868 15487 ts_tablet_manager.cc:1905] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:46.956851 15487 ts_tablet_manager.cc:1918] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:46.957310 15487 log.cc:1196] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/d8a0e4079ebe47aaa3c275ed7f559ab8
I20260806 20:34:46.958922 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet d8a0e4079ebe47aaa3c275ed7f559ab8 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:47.974149 14551 consensus_queue.cc:237] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:47.979566 14633 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:47.979977 14518 consensus_peers.cc:487] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53bd3d0855da421aac4540ae1bfbb8f3. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:47.980195 14478 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:47.981062 15459 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:47.981933 15470 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:47.989513 15415 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:47.991408 14626 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:47.997171 14478 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.007093 14551 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.017773 14626 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:48.017565 14368 catalog_manager.cc:5495] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.019204 15470 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:34:48.024606 14518 consensus_peers.cc:487] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8986697653e048048cd32eb27e4ae3d4. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:48.027870 14478 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:48.029202 15459 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:48.033035 14633 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.032835 15470 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.054697 14626 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.056782 14478 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.069309 14372 catalog_manager.cc:5495] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.080962 14476 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:48.081375 14595 consensus_peers.cc:487] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3044f79ab3f74f788d8a263e141674a8. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:48.082248 15398 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:48.093132 14551 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:48.094904 15398 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:48.100915 15488 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.102653 14476 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.108243 14551 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.115986 14626 consensus_queue.cc:237] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.126150 14478 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:48.127676 15478 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:34:48.136907 14595 consensus_peers.cc:487] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 568833ac14f74d16a95723bbfa721a96. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:48.137382 14551 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:48.139627 14372 catalog_manager.cc:5495] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.144372 15504 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:48.159961 15398 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.166074 14478 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.198572 14551 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:48.212881 14372 catalog_manager.cc:5495] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:48.417445 15507 ts_tablet_manager.cc:927] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:48.420287 15507 tablet_copy_client.cc:316] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:48.422777 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 8986697653e048048cd32eb27e4ae3d4 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:48.423286 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 8986697653e048048cd32eb27e4ae3d4 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-8986697653e048048cd32eb27e4ae3d4
I20260806 20:34:48.432610 14567 tablet_copy_source_session.cc:205] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:48.435250 15507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8986697653e048048cd32eb27e4ae3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:48.450559 15507 tablet_copy_client.cc:791] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:48.451128 15507 tablet_copy_client.cc:655] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:48.470151 15507 tablet_copy_client.cc:531] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:48.479842 15507 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:48.506925 15507 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:48.507723 15507 tablet_bootstrap.cc:492] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:48.508358 15507 ts_tablet_manager.cc:1397] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260806 20:34:48.510500 15507 raft_consensus.cc:348] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.511358 15507 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:48.512147 15507 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.516757 15507 ts_tablet_manager.cc:1428] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20260806 20:34:48.518748 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-8986697653e048048cd32eb27e4ae3d4 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:48.519124 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-8986697653e048048cd32eb27e4ae3d4 on tablet 8986697653e048048cd32eb27e4ae3d4 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:48.585801 15507 ts_tablet_manager.cc:927] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:48.588729 15507 tablet_copy_client.cc:316] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:48.590163 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 53bd3d0855da421aac4540ae1bfbb8f3 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:48.590721 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 53bd3d0855da421aac4540ae1bfbb8f3 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-53bd3d0855da421aac4540ae1bfbb8f3
I20260806 20:34:48.604612 14567 tablet_copy_source_session.cc:205] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:48.607260 15507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bd3d0855da421aac4540ae1bfbb8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:48.627722 15507 tablet_copy_client.cc:791] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:48.628345 15507 tablet_copy_client.cc:655] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:48.630653 15513 ts_tablet_manager.cc:927] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:48.632503 15507 tablet_copy_client.cc:531] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:48.639910 15513 tablet_copy_client.cc:316] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:48.641633 14643 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet 568833ac14f74d16a95723bbfa721a96 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:48.642287 15507 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:48.642176 14643 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet 568833ac14f74d16a95723bbfa721a96 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-568833ac14f74d16a95723bbfa721a96
I20260806 20:34:48.649925 14643 tablet_copy_source_session.cc:205] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:48.653627 15513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568833ac14f74d16a95723bbfa721a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:48.680660 15513 tablet_copy_client.cc:791] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:48.681353 15513 tablet_copy_client.cc:655] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:48.684653 15517 ts_tablet_manager.cc:927] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:48.685403 15507 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:48.686343 15513 tablet_copy_client.cc:531] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:48.687439 15507 tablet_bootstrap.cc:492] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:48.688437 15507 ts_tablet_manager.cc:1397] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.012s
I20260806 20:34:48.691256 15507 raft_consensus.cc:348] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.692224 15507 raft_consensus.cc:729] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:48.692950 15507 consensus_queue.cc:260] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.695643 15507 ts_tablet_manager.cc:1428] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260806 20:34:48.697955 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-53bd3d0855da421aac4540ae1bfbb8f3 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:48.698338 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-53bd3d0855da421aac4540ae1bfbb8f3 on tablet 53bd3d0855da421aac4540ae1bfbb8f3 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:48.706082 15513 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:48.708904 15517 tablet_copy_client.cc:316] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.131:45209
I20260806 20:34:48.710430 14643 tablet_copy_service.cc:140] P df1245ecd2034e11ae6cf688c1c10a4d: Received BeginTabletCopySession request for tablet 3044f79ab3f74f788d8a263e141674a8 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:51240)
I20260806 20:34:48.710987 14643 tablet_copy_service.cc:161] P df1245ecd2034e11ae6cf688c1c10a4d: Beginning new tablet copy session on tablet 3044f79ab3f74f788d8a263e141674a8 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:51240: session id = 148bf2a11dba423ab364371f0458e871-3044f79ab3f74f788d8a263e141674a8
I20260806 20:34:48.719578 14643 tablet_copy_source_session.cc:205] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:48.722736 15517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3044f79ab3f74f788d8a263e141674a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:48.727020 15513 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:48.729079 15513 tablet_bootstrap.cc:492] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:48.729779 15513 ts_tablet_manager.cc:1397] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20260806 20:34:48.732640 15513 raft_consensus.cc:348] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.733500 15513 raft_consensus.cc:729] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:48.734127 15513 consensus_queue.cc:260] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.736547 15513 ts_tablet_manager.cc:1428] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 20:34:48.752753 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-568833ac14f74d16a95723bbfa721a96 received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:48.755153 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-568833ac14f74d16a95723bbfa721a96 on tablet 568833ac14f74d16a95723bbfa721a96 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:48.765293 15517 tablet_copy_client.cc:791] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:48.766026 15517 tablet_copy_client.cc:655] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:48.770040 15517 tablet_copy_client.cc:531] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:48.778167 15517 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:48.795704 15517 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:48.796674 15517 tablet_bootstrap.cc:492] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:48.797344 15517 ts_tablet_manager.cc:1397] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20260806 20:34:48.799634 15517 raft_consensus.cc:348] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.800371 15517 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:48.801086 15517 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:48.802906 15517 ts_tablet_manager.cc:1428] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 20:34:48.805442 14643 tablet_copy_service.cc:342] P df1245ecd2034e11ae6cf688c1c10a4d: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-3044f79ab3f74f788d8a263e141674a8 received from {username='slave'} at 127.0.0.1:51240
I20260806 20:34:48.805828 14643 tablet_copy_service.cc:433] P df1245ecd2034e11ae6cf688c1c10a4d: ending tablet copy session 148bf2a11dba423ab364371f0458e871-3044f79ab3f74f788d8a263e141674a8 on tablet 3044f79ab3f74f788d8a263e141674a8 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:48.990444 15186 raft_consensus.cc:1203] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:49.019312 15186 raft_consensus.cc:1203] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:49.156564 15187 raft_consensus.cc:1203] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:49.283521 15187 raft_consensus.cc:1203] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:49.490787 15470 raft_consensus.cc:1050] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:49.492957 15470 consensus_queue.cc:237] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:49.499153 14633 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.499581 15187 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.500679 15459 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:49.499636 14478 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.501761 15524 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:49.502882 15470 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:49.520107 15492 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.522131 15187 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.522320 14478 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.537267 14372 catalog_manager.cc:5495] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.546877 14633 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.565938 15459 raft_consensus.cc:1050] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:49.568703 15459 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:49.586428 15187 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.588111 14633 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.589223 14476 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.589491 15529 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:49.590751 15470 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:49.599591 14552 consensus_queue.cc:237] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:49.600867 15459 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:49.610293 15524 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.623770 15186 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:49.625800 14471 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:49.627744 15470 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:49.629534 15492 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:49.634402 14625 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.646214 15184 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.648785 15529 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.655498 14476 raft_consensus.cc:2932] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.655561 14471 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.681571 14370 catalog_manager.cc:5495] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.684520 15504 raft_consensus.cc:1050] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:49.689260 15504 consensus_queue.cc:237] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:49.700809 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bd3d0855da421aac4540ae1bfbb8f3 with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:49.706166 15540 tablet_replica.cc:330] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:49.712633 15184 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.716042 14549 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.711169 14370 catalog_manager.cc:5495] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:49.714504 14478 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.718902 15478 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:34:49.722043 15488 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:49.723438 15504 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:49.707708 15540 raft_consensus.cc:2229] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:49.731918 15540 raft_consensus.cc:2258] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:49.753352 15540 ts_tablet_manager.cc:1905] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:49.760399 15187 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.762423 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 53bd3d0855da421aac4540ae1bfbb8f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:49.777432 15504 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.785104 14552 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:49.791124 15187 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:49.792434 15537 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:49.796384 14633 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:49.807713 15537 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:49.807101 14476 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.810529 15538 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.813877 15186 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.815240 14552 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.830986 15540 ts_tablet_manager.cc:1918] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:49.831413 15540 log.cc:1196] T 53bd3d0855da421aac4540ae1bfbb8f3 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/53bd3d0855da421aac4540ae1bfbb8f3
I20260806 20:34:49.832973 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet 53bd3d0855da421aac4540ae1bfbb8f3 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:49.846098 14368 catalog_manager.cc:5495] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.849151 14633 raft_consensus.cc:2932] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.861321 14372 catalog_manager.cc:5495] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:49.862856 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 8986697653e048048cd32eb27e4ae3d4 with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
I20260806 20:34:49.876569 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8986697653e048048cd32eb27e4ae3d4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:49.894644 15478 raft_consensus.cc:1050] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:49.903038 15478 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:49.911252 14627 consensus_queue.cc:237] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:49.892647 15184 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.912930 15546 tablet_replica.cc:330] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:49.944636 15186 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:49.937603 15546 raft_consensus.cc:2229] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:49.948693 15488 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:49.965797 15546 raft_consensus.cc:2258] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:49.962635 15542 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.967705 15184 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:49.984611 15546 ts_tablet_manager.cc:1905] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:49.988035 14552 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:49.988436 15186 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.990296 15547 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:49.991923 14549 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:49.991568 15541 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 20:34:50.000613 14552 raft_consensus.cc:2932] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.000280 15478 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:50.004369 14476 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:50.008353 15478 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:50.024271 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 568833ac14f74d16a95723bbfa721a96 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:50.033053 14369 catalog_manager.cc:5495] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.077152 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 568833ac14f74d16a95723bbfa721a96 with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
I20260806 20:34:50.079761 15546 ts_tablet_manager.cc:1918] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:50.080526 15546 log.cc:1196] T 8986697653e048048cd32eb27e4ae3d4 P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/8986697653e048048cd32eb27e4ae3d4
I20260806 20:34:50.083236 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet 8986697653e048048cd32eb27e4ae3d4 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:50.083249 15546 tablet_replica.cc:330] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:50.085120 15546 raft_consensus.cc:2229] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:50.086498 15546 raft_consensus.cc:2258] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:50.090075 14476 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.087374 15541 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.092779 14549 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.096395 15546 ts_tablet_manager.cc:1905] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:50.126371 15186 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.140916 14368 catalog_manager.cc:5495] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.142225 15546 ts_tablet_manager.cc:1918] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:50.143067 15546 log.cc:1196] T 568833ac14f74d16a95723bbfa721a96 P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/568833ac14f74d16a95723bbfa721a96
I20260806 20:34:50.145555 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet 568833ac14f74d16a95723bbfa721a96 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:50.174540 14625 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:50.179860 15186 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:50.181648 14478 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:50.181813 15547 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:50.183342 15478 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:50.192206 15542 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.194087 14476 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.195583 15186 raft_consensus.cc:2932] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.208946 14359 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3044f79ab3f74f788d8a263e141674a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:50.213423 14372 catalog_manager.cc:5495] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:50.237803 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 3044f79ab3f74f788d8a263e141674a8 with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:50.241355 15554 tablet_replica.cc:330] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:50.243098 15554 raft_consensus.cc:2229] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:50.243966 15554 raft_consensus.cc:2258] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:50.248996 15554 ts_tablet_manager.cc:1905] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:50.271849 15554 ts_tablet_manager.cc:1918] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:50.272274 15554 log.cc:1196] T 3044f79ab3f74f788d8a263e141674a8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/3044f79ab3f74f788d8a263e141674a8
I20260806 20:34:50.274081 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet 3044f79ab3f74f788d8a263e141674a8 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:51.298844 14549 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
W20260806 20:34:51.305341 14518 consensus_peers.cc:487] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 289df1801e0045beb52c1aaaf43e73f0. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:51.305532 14627 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:51.306753 15559 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:51.307137 14476 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:51.308283 15459 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:51.316319 15537 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.317636 14476 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.319525 14627 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.335268 14549 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:51.336151 14372 catalog_manager.cc:5495] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:51.343336 14625 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:51.345440 14471 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:51.346951 15562 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:51.349090 15459 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:34:51.352144 14518 consensus_peers.cc:487] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4248ec88bd4b44e990938a74fe3c4fdc. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:51.371138 15537 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.372797 14625 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.375340 14471 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.389142 14476 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:51.391165 14370 catalog_manager.cc:5495] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.408020 14627 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:51.409029 14441 consensus_peers.cc:487] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d5ff5043e9e4bdbb65a2354b475a2c1. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:51.409632 15556 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:51.413599 14552 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:51.415534 15556 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:51.427374 15522 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.429112 14552 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.430424 14627 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.448828 14370 catalog_manager.cc:5495] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.454021 14552 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:51.464846 14627 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:51.465301 14518 consensus_peers.cc:487] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8844659c5dd24572bab900a68fc14df6. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:51.466192 15559 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:51.467706 14471 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:51.469219 15559 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:51.475271 15459 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.482319 14625 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.495230 14370 catalog_manager.cc:5495] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:51.501307 14471 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:51.769950 15577 ts_tablet_manager.cc:927] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:51.773452 15577 tablet_copy_client.cc:316] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:51.774931 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 289df1801e0045beb52c1aaaf43e73f0 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:51.775445 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 289df1801e0045beb52c1aaaf43e73f0 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-289df1801e0045beb52c1aaaf43e73f0
I20260806 20:34:51.784834 14567 tablet_copy_source_session.cc:205] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:51.787566 15579 ts_tablet_manager.cc:927] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:51.824645 15577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289df1801e0045beb52c1aaaf43e73f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:51.869242 15579 tablet_copy_client.cc:316] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:51.886793 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:51.887688 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-8d5ff5043e9e4bdbb65a2354b475a2c1
I20260806 20:34:51.896641 15577 tablet_copy_client.cc:791] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:51.898746 15577 tablet_copy_client.cc:655] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:51.905449 15577 tablet_copy_client.cc:531] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:51.907431 15587 ts_tablet_manager.cc:927] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:34:51.908420 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 52374.3us
I20260806 20:34:51.914893 14489 tablet_copy_source_session.cc:205] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:51.925778 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:51.943907 15577 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:51.945904 15587 tablet_copy_client.cc:316] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:51.948485 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 4248ec88bd4b44e990938a74fe3c4fdc from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:51.949334 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 4248ec88bd4b44e990938a74fe3c4fdc from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-4248ec88bd4b44e990938a74fe3c4fdc
I20260806 20:34:51.971998 14567 tablet_copy_source_session.cc:205] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:51.972471 15579 tablet_copy_client.cc:791] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:51.979405 15579 tablet_copy_client.cc:655] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:51.979909 15587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4248ec88bd4b44e990938a74fe3c4fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:51.986125 15579 tablet_copy_client.cc:531] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:52.011286 15577 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:52.012429 15577 tablet_bootstrap.cc:492] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:52.013230 15577 ts_tablet_manager.cc:1397] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.070s user 0.020s sys 0.025s
I20260806 20:34:52.014909 15579 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:52.016690 15577 raft_consensus.cc:348] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.017305 15587 tablet_copy_client.cc:791] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:52.017784 15577 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:52.018121 15587 tablet_copy_client.cc:655] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:52.019595 15577 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.023379 15587 tablet_copy_client.cc:531] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:52.039479 15587 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:52.042786 15577 ts_tablet_manager.cc:1428] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.029s user 0.008s sys 0.018s
I20260806 20:34:52.050015 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-289df1801e0045beb52c1aaaf43e73f0 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:52.050428 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-289df1801e0045beb52c1aaaf43e73f0 on tablet 289df1801e0045beb52c1aaaf43e73f0 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:52.055384 15577 ts_tablet_manager.cc:927] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:52.068922 15577 tablet_copy_client.cc:316] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:52.071130 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 8844659c5dd24572bab900a68fc14df6 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:52.071669 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 8844659c5dd24572bab900a68fc14df6 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-8844659c5dd24572bab900a68fc14df6
I20260806 20:34:52.079861 14567 tablet_copy_source_session.cc:205] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:52.080538 15587 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:52.081646 15587 tablet_bootstrap.cc:492] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:52.082362 15587 ts_tablet_manager.cc:1397] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.043s user 0.029s sys 0.001s
I20260806 20:34:52.083338 15577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8844659c5dd24572bab900a68fc14df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:52.085794 15587 raft_consensus.cc:348] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.086809 15587 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:52.087983 15579 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:52.087532 15587 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.088980 15579 tablet_bootstrap.cc:492] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:52.089777 15579 ts_tablet_manager.cc:1397] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.075s user 0.018s sys 0.014s
I20260806 20:34:52.090525 15587 ts_tablet_manager.cc:1428] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:34:52.092751 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-4248ec88bd4b44e990938a74fe3c4fdc received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:52.093173 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-4248ec88bd4b44e990938a74fe3c4fdc on tablet 4248ec88bd4b44e990938a74fe3c4fdc with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:52.094254 15579 raft_consensus.cc:348] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.095470 15579 raft_consensus.cc:729] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:52.096190 15579 consensus_queue.cc:260] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.098438 15579 ts_tablet_manager.cc:1428] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 20:34:52.099900 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-8d5ff5043e9e4bdbb65a2354b475a2c1 received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:52.100270 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-8d5ff5043e9e4bdbb65a2354b475a2c1 on tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:52.133267 15577 tablet_copy_client.cc:791] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:52.133875 15577 tablet_copy_client.cc:655] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:52.137745 15577 tablet_copy_client.cc:531] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:52.146879 15577 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:52.166386 15577 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:52.167235 15577 tablet_bootstrap.cc:492] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:52.167871 15577 ts_tablet_manager.cc:1397] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20260806 20:34:52.169866 15187 raft_consensus.cc:1203] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:52.170527 15577 raft_consensus.cc:348] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.171399 15577 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:52.172076 15577 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:52.175164 15577 ts_tablet_manager.cc:1428] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.007s sys 0.003s
I20260806 20:34:52.177052 14567 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-8844659c5dd24572bab900a68fc14df6 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:52.177393 14567 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-8844659c5dd24572bab900a68fc14df6 on tablet 8844659c5dd24572bab900a68fc14df6 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:52.365291 15187 raft_consensus.cc:1203] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:52.421412 15187 raft_consensus.cc:1203] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:52.650300 15186 raft_consensus.cc:1203] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:52.671423 15597 raft_consensus.cc:1050] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:52.679275 15597 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:52.696934 14627 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:52.701143 15584 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:52.704810 15186 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:52.705053 14479 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:52.707430 15597 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:52.714305 15559 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:52.760150 15559 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.832549 14625 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.844044 14476 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.858819 15187 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.870653 14372 catalog_manager.cc:5495] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:52.905911 14552 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:52.919562 15186 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:52.920281 14470 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:52.921582 15559 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:52.923251 15584 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:52.928391 15559 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.933357 15187 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.933851 14476 raft_consensus.cc:2932] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.946543 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 289df1801e0045beb52c1aaaf43e73f0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:52.949787 14370 catalog_manager.cc:5495] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:52.960327 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet 289df1801e0045beb52c1aaaf43e73f0 with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:52.970077 15610 tablet_replica.cc:330] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:52.972769 15610 raft_consensus.cc:2229] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:52.973544 15610 raft_consensus.cc:2258] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:52.981688 15610 ts_tablet_manager.cc:1905] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:52.981812 15556 raft_consensus.cc:1050] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:52.997867 15556 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:53.001959 15610 ts_tablet_manager.cc:1918] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:53.002626 15610 log.cc:1196] T 289df1801e0045beb52c1aaaf43e73f0 P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/289df1801e0045beb52c1aaaf43e73f0
I20260806 20:34:53.006470 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet 289df1801e0045beb52c1aaaf43e73f0 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:53.011982 15583 raft_consensus.cc:1050] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:53.015265 15187 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.014905 15583 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:53.017817 14625 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.018200 14550 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.021046 15573 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:53.021992 15556 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:34:53.027190 14625 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.028682 15559 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:53.030965 15522 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:53.033663 15186 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.036444 14470 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.038002 15559 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:53.039968 15583 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:53.053797 14551 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.052224 15556 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.069146 14625 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.091949 14370 catalog_manager.cc:5495] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.098016 15559 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.137951 14470 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.139632 15187 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.155495 14627 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.160909 15583 raft_consensus.cc:1050] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:53.163534 15583 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:53.173589 14628 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.174722 15187 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.177331 14471 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:53.177911 15559 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:53.180703 15584 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:53.183321 15583 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:53.185240 15187 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.210879 14370 catalog_manager.cc:5495] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.241039 14471 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:53.249908 15600 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.255687 15187 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:53.255649 15186 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.260648 15613 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:53.261441 14633 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:53.269546 15573 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.280860 15613 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:53.288537 14627 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.289649 15186 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.301649 14369 catalog_manager.cc:5495] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:34:53.317350 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47942.2us
I20260806 20:34:53.318645 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:53.318934 14552 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:53.305225 14471 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.337343 14627 raft_consensus.cc:2932] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.339473 15187 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:53.340969 15600 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:53.342509 14479 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:53.342126 14372 catalog_manager.cc:5495] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 reported cstate change: config changed from index 3 to 4, VOTER 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) evicted. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.344827 15584 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:53.352133 15583 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.355232 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 with delete_type TABLET_DATA_TOMBSTONED (TS 8e0cfd35f2994f91aaae46c25c6d6bff not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54486
I20260806 20:34:53.357600 14552 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:53.363906 14471 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.372550 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4248ec88bd4b44e990938a74fe3c4fdc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:53.379940 14633 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:53.380546 15187 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:34:53.381722 15583 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:53.385619 15584 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:53.389878 15559 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.408226 14370 catalog_manager.cc:5495] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:53.415678 15626 tablet_replica.cc:330] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: stopping tablet replica
I20260806 20:34:53.416919 15626 raft_consensus.cc:2229] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:53.417682 15626 raft_consensus.cc:2258] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:53.420490 14356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8844659c5dd24572bab900a68fc14df6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:34:53.424638 15626 ts_tablet_manager.cc:1905] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:53.428357 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet 4248ec88bd4b44e990938a74fe3c4fdc with delete_type TABLET_DATA_TOMBSTONED (TS df1245ecd2034e11ae6cf688c1c10a4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38498
I20260806 20:34:53.429347 15610 tablet_replica.cc:330] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: stopping tablet replica
I20260806 20:34:53.430331 15610 raft_consensus.cc:2229] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:53.431066 15610 raft_consensus.cc:2258] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:53.430497 14627 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.434208 15610 ts_tablet_manager.cc:1905] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:53.439692 14370 catalog_manager.cc:5495] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:53.452849 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 8844659c5dd24572bab900a68fc14df6 with delete_type TABLET_DATA_TOMBSTONED (TS 92828dd6dbe74f29a04d4e702b502621 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54494
W20260806 20:34:53.454470 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 61252.4us
I20260806 20:34:53.456971 15186 raft_consensus.cc:2932] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.458078 15610 ts_tablet_manager.cc:1918] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:53.458616 15610 log.cc:1196] T 4248ec88bd4b44e990938a74fe3c4fdc P df1245ecd2034e11ae6cf688c1c10a4d: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-2-root/wals/4248ec88bd4b44e990938a74fe3c4fdc
I20260806 20:34:53.460315 14359 catalog_manager.cc:4841] TS df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): tablet 4248ec88bd4b44e990938a74fe3c4fdc (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:53.460912 15630 tablet_replica.cc:330] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: stopping tablet replica
I20260806 20:34:53.462036 15630 raft_consensus.cc:2229] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:34:53.462785 15630 raft_consensus.cc:2258] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:34:53.467397 15630 ts_tablet_manager.cc:1905] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:34:53.476231 15626 ts_tablet_manager.cc:1918] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:53.476823 15626 log.cc:1196] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 8e0cfd35f2994f91aaae46c25c6d6bff: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-1-root/wals/8d5ff5043e9e4bdbb65a2354b475a2c1
I20260806 20:34:53.476536 15187 raft_consensus.cc:2932] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:53.478971 14356 catalog_manager.cc:4841] TS 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): tablet 8d5ff5043e9e4bdbb65a2354b475a2c1 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:53.483923 15630 ts_tablet_manager.cc:1918] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:34:53.484741 15630 log.cc:1196] T 8844659c5dd24572bab900a68fc14df6 P 92828dd6dbe74f29a04d4e702b502621: Deleting WAL directory at /tmp/dist-test-taskoy95tQ/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786048401793185-13282-0/minicluster-data/ts-0-root/wals/8844659c5dd24572bab900a68fc14df6
I20260806 20:34:53.486703 14355 catalog_manager.cc:4841] TS 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): tablet 8844659c5dd24572bab900a68fc14df6 (table test-workload [id=8a9bb69966b34eb8b32423b196259815]) successfully deleted
I20260806 20:34:54.502668 14471 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:54.509083 14633 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:34:54.509882 14441 consensus_peers.cc:487] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a94bd288a57145e8b254611b1dd82586. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:54.510594 15633 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:54.514443 14551 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:54.516772 15633 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:54.518993 15633 raft_consensus.cc:2932] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.520537 14633 raft_consensus.cc:2932] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.527001 14550 raft_consensus.cc:2932] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.535234 14552 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:54.539505 14370 catalog_manager.cc:5495] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:34:54.551429 14518 consensus_peers.cc:487] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 500956bd1a3846a3a7b16c51ddecbc44. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:54.553064 14478 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:54.554596 15631 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:54.556753 14633 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:34:54.561388 15631 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:54.565209 15628 raft_consensus.cc:2932] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.566906 14478 raft_consensus.cc:2932] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.599474 14478 consensus_queue.cc:237] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:54.605746 14633 raft_consensus.cc:2932] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.607049 14369 catalog_manager.cc:5495] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 3, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:34:54.629386 14441 consensus_peers.cc:487] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b16d0272c85445ed95a93660f8851656. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:54.629966 14633 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:54.631305 15624 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:54.633816 14552 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:54.635267 15624 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:54.640909 15633 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.642568 14633 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.647125 14552 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.653813 14550 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
W20260806 20:34:54.661569 14518 consensus_peers.cc:487] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff -> Peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Couldn't send request to peer 148bf2a11dba423ab364371f0458e871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f41753baced49a1997940dfc28b2e00. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:34:54.662410 14633 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:54.663918 15628 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:54.666303 14478 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:34:54.667042 14369 catalog_manager.cc:5495] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.670415 15628 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:54.676368 15643 raft_consensus.cc:2932] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.678107 14628 raft_consensus.cc:2932] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.698263 14370 catalog_manager.cc:5495] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.699301 14471 raft_consensus.cc:2932] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 148bf2a11dba423ab364371f0458e871 (127.12.248.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } } }
I20260806 20:34:54.836531 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:54.916126 14551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:54.916915 14551 raft_consensus.cc:595] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:54.919222 14551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:54.919750 14551 raft_consensus.cc:595] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:54.921903 14551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:54.922310 14551 raft_consensus.cc:595] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:54.942610 14628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:34:54.943207 14628 raft_consensus.cc:595] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:54.945717 14551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:54.946195 14551 raft_consensus.cc:595] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:54.948391 14551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5277702c39f453cb1063cf506183484"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:54.948866 14551 raft_consensus.cc:595] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:54.966915 14478 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:34:54.973577 14478 raft_consensus.cc:595] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:54.976578 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260806 20:34:54.978228 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:54.978883 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:54.996345 15628 raft_consensus.cc:979] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:54.996855 15628 raft_consensus.cc:1067] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:54.998338 15187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:54.998920 15187 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:54.999277 15187 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.015064 15628 raft_consensus.cc:979] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:55.015643 15628 raft_consensus.cc:1067] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:55.015437 15187 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.017463 15184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:55.018026 15184 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:55.018383 15184 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.020507 15655 ts_tablet_manager.cc:927] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:55.024652 15599 raft_consensus.cc:979] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:55.025167 15644 raft_consensus.cc:1067] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:55.026613 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:55.027295 14551 raft_consensus.cc:3032] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:55.027738 14551 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:55.028538 14551 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.029198 14478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:54502
I20260806 20:34:55.029626 14551 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.029739 14478 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:55.030103 14478 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.031526 15187 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:55.033488 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:55.034013 14628 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.040948 14551 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:55.042905 15184 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.043136 15151 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:34:55.044301 14628 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:55.046254 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:55.046933 14471 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.047915 15658 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:55.051379 14478 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.054518 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:55.054795 14471 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:55.055156 14628 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.055694 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:55.056244 14552 raft_consensus.cc:3032] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:55.056326 15150 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:55.056770 14552 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:55.059510 14552 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.061630 14628 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:34:55.062553 15184 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:55.063510 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:55.065399 14550 raft_consensus.cc:3032] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:55.065688 14550 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:55.053624 14478 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:55.066892 14444 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:55.068142 15633 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:55.068817 15633 raft_consensus.cc:686] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:55.064146 15657 ts_tablet_manager.cc:927] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789)
I20260806 20:34:55.064549 14552 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.066344 14550 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.069679 15633 consensus_queue.cc:237] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.070904 14550 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.066273 15658 raft_consensus.cc:686] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:34:55.074491 15658 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.079967 14550 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:55.083544 15655 tablet_copy_client.cc:316] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
I20260806 20:34:55.100183 14552 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:34:55.098462 14369 catalog_manager.cc:5495] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:55.125250 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet a94bd288a57145e8b254611b1dd82586 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:55.125870 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet a94bd288a57145e8b254611b1dd82586 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-a94bd288a57145e8b254611b1dd82586
I20260806 20:34:55.143075 14370 catalog_manager.cc:5495] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:55.128719 15658 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:55.167075 14489 tablet_copy_source_session.cc:205] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:55.168290 15658 raft_consensus.cc:686] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:34:55.208710 15657 tablet_copy_client.cc:316] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.129:44789
W20260806 20:34:55.214380 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 69702.3us
I20260806 20:34:55.215322 15655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94bd288a57145e8b254611b1dd82586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:55.216219 14489 tablet_copy_service.cc:140] P 92828dd6dbe74f29a04d4e702b502621: Received BeginTabletCopySession request for tablet b16d0272c85445ed95a93660f8851656 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:35910)
I20260806 20:34:55.216789 14489 tablet_copy_service.cc:161] P 92828dd6dbe74f29a04d4e702b502621: Beginning new tablet copy session on tablet b16d0272c85445ed95a93660f8851656 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:35910: session id = 148bf2a11dba423ab364371f0458e871-b16d0272c85445ed95a93660f8851656
I20260806 20:34:55.224006 15585 raft_consensus.cc:979] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d: : Instructing follower 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:34:55.224607 15585 raft_consensus.cc:1067] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Signalling peer 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:34:55.212284 15658 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.229281 15667 ts_tablet_manager.cc:927] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:55.230294 14489 tablet_copy_source_session.cc:205] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:55.232543 14548 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:54508
I20260806 20:34:55.242801 14548 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:55.257607 14548 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.279819 15672 ts_tablet_manager.cc:927] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Initiating tablet copy from peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:55.282735 15657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16d0272c85445ed95a93660f8851656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:55.314983 14548 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.318221 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:55.318804 14478 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.323491 15669 raft_consensus.cc:979] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:55.324054 15628 raft_consensus.cc:1067] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:55.323200 14370 catalog_manager.cc:5495] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:55.325574 15184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:55.325994 15655 tablet_copy_client.cc:791] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:55.326114 15184 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:55.326628 15655 tablet_copy_client.cc:655] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:55.326656 15184 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.324409 15667 tablet_copy_client.cc:316] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:55.328608 14567 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 500956bd1a3846a3a7b16c51ddecbc44 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:55.329114 14567 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 500956bd1a3846a3a7b16c51ddecbc44 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-500956bd1a3846a3a7b16c51ddecbc44
I20260806 20:34:55.332394 15184 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.333990 15184 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:55.334892 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:55.335489 14628 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.335652 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:55.337203 14552 raft_consensus.cc:3032] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:55.337463 14552 raft_consensus.cc:729] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:55.341110 14552 consensus_queue.cc:260] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.342373 14552 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.343618 15657 tablet_copy_client.cc:791] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:55.344178 15657 tablet_copy_client.cc:655] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:55.344602 15672 tablet_copy_client.cc:316] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.130:39545
I20260806 20:34:55.345911 14548 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:55.347750 14478 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:34:55.347668 14566 tablet_copy_service.cc:140] P 8e0cfd35f2994f91aaae46c25c6d6bff: Received BeginTabletCopySession request for tablet 7f41753baced49a1997940dfc28b2e00 from peer 148bf2a11dba423ab364371f0458e871 ({username='slave'} at 127.0.0.1:36504)
I20260806 20:34:55.346719 15655 tablet_copy_client.cc:531] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:55.348735 14566 tablet_copy_service.cc:161] P 8e0cfd35f2994f91aaae46c25c6d6bff: Beginning new tablet copy session on tablet 7f41753baced49a1997940dfc28b2e00 from peer 148bf2a11dba423ab364371f0458e871 at {username='slave'} at 127.0.0.1:36504: session id = 148bf2a11dba423ab364371f0458e871-7f41753baced49a1997940dfc28b2e00
I20260806 20:34:55.350629 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:55.351190 14627 raft_consensus.cc:3032] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:55.351450 14627 raft_consensus.cc:729] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:55.351961 14628 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:55.352187 14627 consensus_queue.cc:260] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.353451 14627 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.353994 15153 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:55.355036 15658 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:55.355486 15658 raft_consensus.cc:686] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:34:55.355693 14567 tablet_copy_source_session.cc:205] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:55.356331 15658 consensus_queue.cc:237] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:55.359042 15667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500956bd1a3846a3a7b16c51ddecbc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:55.355584 14518 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:55.361873 15599 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:55.362334 15599 raft_consensus.cc:686] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:55.363205 15599 consensus_queue.cc:237] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.364539 15655 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:55.368769 14552 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:55.373395 15657 tablet_copy_client.cc:531] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:55.381876 15667 tablet_copy_client.cc:791] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:55.382462 15667 tablet_copy_client.cc:655] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:55.388664 14370 catalog_manager.cc:5495] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:55.389995 15667 tablet_copy_client.cc:531] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:55.395315 14369 catalog_manager.cc:5495] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 2 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:55.392261 14566 tablet_copy_source_session.cc:205] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:34:55.401793 14627 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:34:55.403870 15667 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:55.409926 15657 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:55.411267 15672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f41753baced49a1997940dfc28b2e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:34:55.428158 15655 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:55.436199 15655 tablet_bootstrap.cc:492] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:55.437114 15655 ts_tablet_manager.cc:1397] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.073s user 0.023s sys 0.009s
I20260806 20:34:55.442552 15655 raft_consensus.cc:348] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.445528 15655 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:55.446591 15655 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.450906 15655 ts_tablet_manager.cc:1428] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260806 20:34:55.452977 15657 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:55.453722 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-a94bd288a57145e8b254611b1dd82586 received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:55.454128 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-a94bd288a57145e8b254611b1dd82586 on tablet a94bd288a57145e8b254611b1dd82586 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:55.455555 15657 tablet_bootstrap.cc:492] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:55.456283 15657 ts_tablet_manager.cc:1397] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.050s user 0.028s sys 0.006s
I20260806 20:34:55.462387 15657 raft_consensus.cc:348] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.463315 15657 raft_consensus.cc:729] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:55.463913 15657 consensus_queue.cc:260] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.466547 15657 ts_tablet_manager.cc:1428] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 20:34:55.468896 14489 tablet_copy_service.cc:342] P 92828dd6dbe74f29a04d4e702b502621: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-b16d0272c85445ed95a93660f8851656 received from {username='slave'} at 127.0.0.1:35910
I20260806 20:34:55.469285 14489 tablet_copy_service.cc:433] P 92828dd6dbe74f29a04d4e702b502621: ending tablet copy session 148bf2a11dba423ab364371f0458e871-b16d0272c85445ed95a93660f8851656 on tablet b16d0272c85445ed95a93660f8851656 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:55.474439 15573 raft_consensus.cc:979] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: : Instructing follower 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:34:55.475133 15665 raft_consensus.cc:1067] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Signalling peer 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:34:55.476893 14548 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:54498
I20260806 20:34:55.477648 14548 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:55.478190 14548 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.493680 14548 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.496821 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:55.497223 14548 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:55.497395 14478 raft_consensus.cc:3032] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:55.497520 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:55.497880 14478 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:55.498111 14627 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.498852 14478 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.500311 14478 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.505107 14627 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:34:55.509697 14478 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:34:55.510735 15672 tablet_copy_client.cc:791] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 0 data blocks...
I20260806 20:34:55.511297 15672 tablet_copy_client.cc:655] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: tablet copy: Starting download of 1 WAL segments...
I20260806 20:34:55.511205 14521 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:55.512070 15599 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:55.512560 15599 raft_consensus.cc:686] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:55.513404 15599 consensus_queue.cc:237] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.515278 15672 tablet_copy_client.cc:531] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:34:55.522668 15628 raft_consensus.cc:979] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:55.523187 15628 raft_consensus.cc:1067] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:55.524802 14478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5277702c39f453cb1063cf506183484"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:54502
I20260806 20:34:55.525327 14478 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:55.525653 14478 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.527926 15672 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Bootstrap starting.
I20260806 20:34:55.530208 14627 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:55.532505 15667 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:55.534062 15667 tablet_bootstrap.cc:492] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:55.534737 15667 ts_tablet_manager.cc:1397] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.133s user 0.042s sys 0.032s
I20260806 20:34:55.537047 15658 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:34:55.542162 15184 raft_consensus.cc:1203] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:55.553668 15667 raft_consensus.cc:348] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.560111 15667 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:55.561054 15667 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.563454 15667 ts_tablet_manager.cc:1428] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260806 20:34:55.572685 15186 raft_consensus.cc:1203] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:55.575889 14566 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-500956bd1a3846a3a7b16c51ddecbc44 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:55.576334 14566 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-500956bd1a3846a3a7b16c51ddecbc44 on tablet 500956bd1a3846a3a7b16c51ddecbc44 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:55.584471 15186 raft_consensus.cc:1203] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20260806 20:34:55.588902 15672 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:34:55.589811 15672 tablet_bootstrap.cc:492] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Bootstrap complete.
I20260806 20:34:55.590463 15672 ts_tablet_manager.cc:1397] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Time spent bootstrapping tablet: real 0.063s user 0.022s sys 0.003s
I20260806 20:34:55.593401 15672 raft_consensus.cc:348] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.594790 15672 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Initialized, Role: LEARNER
I20260806 20:34:55.595438 15672 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:55.597875 15672 ts_tablet_manager.cc:1428] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:34:55.601769 14370 catalog_manager.cc:5495] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 2 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:55.606225 14566 tablet_copy_service.cc:342] P 8e0cfd35f2994f91aaae46c25c6d6bff: Request end of tablet copy session 148bf2a11dba423ab364371f0458e871-7f41753baced49a1997940dfc28b2e00 received from {username='slave'} at 127.0.0.1:36504
I20260806 20:34:55.606657 14566 tablet_copy_service.cc:433] P 8e0cfd35f2994f91aaae46c25c6d6bff: ending tablet copy session 148bf2a11dba423ab364371f0458e871-7f41753baced49a1997940dfc28b2e00 on tablet 7f41753baced49a1997940dfc28b2e00 with peer 148bf2a11dba423ab364371f0458e871
I20260806 20:34:55.619871 14552 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:55.625566 14478 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.627733 14478 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:55.629195 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:55.629794 14552 raft_consensus.cc:3032] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:55.630069 14552 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:55.630663 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:55.631220 14633 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.630806 14552 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.631448 15659 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:34:55.631909 14552 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:55.630664 14627 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:55.639953 14552 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:34:55.641829 14442 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:55.642788 15624 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:55.643668 15665 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:55.644572 15665 raft_consensus.cc:686] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:55.645430 15665 consensus_queue.cc:237] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:55.650653 14633 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:34:55.651614 14548 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:55.655223 15665 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:55.667418 14372 catalog_manager.cc:5495] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:55.689952 14471 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:55.691730 15659 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:34:55.742758 14552 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:55.744160 15661 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:34:55.758054 14627 raft_consensus.cc:1261] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:55.759675 15599 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:55.773100 14471 raft_consensus.cc:1261] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:55.774370 15599 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:55.829427 15186 raft_consensus.cc:1203] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:34:55.902415 14633 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:55.904268 15658 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:34:55.905568 14551 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:55.908672 15658 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:34:56.037177 14552 raft_consensus.cc:1261] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:56.038440 15665 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:34:56.055095 14627 raft_consensus.cc:1261] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:56.056617 14470 raft_consensus.cc:1261] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:56.058328 15665 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:56.058517 15599 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:56.075371 15628 raft_consensus.cc:1050] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:56.088616 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:56.106285 15692 raft_consensus.cc:1050] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:56.110280 15692 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:56.079965 15628 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:56.128831 14633 raft_consensus.cc:1261] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:56.132552 15624 raft_consensus.cc:1050] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:56.132714 15643 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:56.303921 15624 consensus_queue.cc:237] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:56.414989 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:56.445659 14633 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:56.448724 15573 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:56.444196 14548 raft_consensus.cc:595] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:56.485313 14470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:34:56.486461 14470 raft_consensus.cc:595] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:56.497499 14470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:34:56.498627 14470 raft_consensus.cc:595] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:56.503248 14627 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:34:56.503885 14627 raft_consensus.cc:595] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:56.512799 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:56.513909 14548 raft_consensus.cc:595] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:56.518230 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:56.520305 14548 raft_consensus.cc:595] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:56.521941 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 20:34:56.524098 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:56.524729 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:56.548395 15187 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 20:34:56.558496 14625 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:56.558765 14628 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260806 20:34:56.561667 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 154186us
I20260806 20:34:56.570099 15184 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:56.600283 14551 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:56.599005 15665 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:34:56.602797 14552 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:56.616202 14470 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 20:34:56.613966 15585 raft_consensus.cc:979] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: : Instructing follower 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:56.620759 15585 raft_consensus.cc:1067] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Signalling peer 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:56.643610 15187 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:56.650061 15712 raft_consensus.cc:979] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:56.650808 15710 raft_consensus.cc:1067] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:56.655056 14478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:54500
I20260806 20:34:56.655905 14478 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:56.656349 14478 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.658182 15712 raft_consensus.cc:979] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:56.659024 15712 raft_consensus.cc:1067] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:56.666898 14478 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:56.661121 15599 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:56.668809 15705 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:34:56.678248 15573 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:34:56.682973 15187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:56.683723 15187 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:56.685316 15573 raft_consensus.cc:979] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:56.686065 15573 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:34:56.684350 15187 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.687888 15184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:56.688627 15184 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:56.689092 15184 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.696295 14478 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:56.718407 15187 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:56.718752 15184 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:56.723606 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:56.724988 14626 raft_consensus.cc:3032] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:56.698184 15706 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260806 20:34:56.725289 14626 raft_consensus.cc:729] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:34:56.726272 14626 consensus_queue.cc:260] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:56.727629 14626 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.727732 15710 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:34:56.702175 15723 raft_consensus.cc:1067] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:56.732424 15184 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:56.739542 15187 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:34:56.751554 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 75582.4us
I20260806 20:34:56.791447 14626 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:34:56.791957 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 106902us
I20260806 20:34:56.799587 14444 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:56.806804 15187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:59384
I20260806 20:34:56.807420 15187 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:56.807791 15187 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.807978 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:56.808601 14552 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.816339 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:56.817006 14478 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.818403 14552 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:34:56.804494 15633 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:56.831094 15187 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:56.834066 14478 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:56.837606 15187 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:56.833818 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:56.802778 15706 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260806 20:34:56.845584 14471 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.861759 14471 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:56.867041 15187 raft_consensus.cc:1226] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:34:56.884081 15150 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:56.885296 15661 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:56.850625 15633 raft_consensus.cc:686] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:56.894327 15573 consensus_queue.cc:1046] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:34:56.894894 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:34:56.926436 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 250512us
W20260806 20:34:56.931538 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 224952us
I20260806 20:34:56.994011 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:57.002373 14549 raft_consensus.cc:3032] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:57.002714 14549 raft_consensus.cc:729] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:57.021521 14549 consensus_queue.cc:260] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.053115 14549 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:56.943526 15633 consensus_queue.cc:237] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:57.028491 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 142865us
W20260806 20:34:57.074684 15718 log.cc:924] Time spent T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260806 20:34:57.052738 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:57.127112 15661 raft_consensus.cc:686] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:34:57.082823 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:57.217787 14626 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:57.218065 15661 consensus_queue.cc:237] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:34:57.129767 15731 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260806 20:34:57.225234 14547 raft_consensus.cc:3032] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:57.078281 15150 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:34:57.232606 14547 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:34:57.233201 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 156946us
I20260806 20:34:57.234246 15739 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:57.233947 14547 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.235776 15739 raft_consensus.cc:686] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:34:57.241196 14368 catalog_manager.cc:5495] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:34:57.262316 14549 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:57.236014 14547 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:57.236796 15739 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.273371 15709 raft_consensus.cc:2932] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.276149 15643 raft_consensus.cc:1050] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: attempting to promote NON_VOTER 148bf2a11dba423ab364371f0458e871 to VOTER
I20260806 20:34:57.290324 15184 raft_consensus.cc:2932] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.279762 15643 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.312534 14471 raft_consensus.cc:3032] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:57.313184 14471 raft_consensus.cc:729] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:34:57.314236 14471 consensus_queue.cc:260] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.316177 14471 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:57.317703 15628 raft_consensus.cc:979] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:57.318337 15628 raft_consensus.cc:1067] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:57.324640 14547 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:57.342099 14626 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
W20260806 20:34:57.336232 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 229246us
I20260806 20:34:57.356428 15153 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:57.357831 15661 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:34:57.358592 15661 raft_consensus.cc:686] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:34:57.360138 15661 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.364591 15184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:57.383699 14471 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:57.384208 15184 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:57.385612 15184 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:57.420717 14633 raft_consensus.cc:2932] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.424325 14550 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.450385 15743 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:34:57.451447 15743 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
W20260806 20:34:57.483680 15716 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20260806 20:34:57.497650 15751 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 20:34:57.486912 15186 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.366607 15710 raft_consensus.cc:2932] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.490417 15724 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:34:57.537472 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 118653us
I20260806 20:34:57.542620 14633 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 20:34:57.561510 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 170326us
I20260806 20:34:57.561594 15184 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.564006 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:57.565958 15184 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:57.537590 15743 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:57.587837 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:57.592516 14633 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:57.602021 15727 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260806 20:34:57.703758 14633 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:57.710947 14474 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:57.713302 15153 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:34:57.717911 15740 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 20:34:57.733409 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 335496us
I20260806 20:34:57.747835 15738 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260806 20:34:57.763221 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:34:57.765645 14625 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:57.764147 14547 raft_consensus.cc:2932] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.767228 14626 raft_consensus.cc:2932] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.777067 14444 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:34:57.779719 14633 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:57.797453 15725 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:34:57.801402 15691 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
W20260806 20:34:57.813578 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 395304us
I20260806 20:34:57.816293 14478 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:34:57.831455 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 240149us
W20260806 20:34:57.844491 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 273189us
W20260806 20:34:57.868819 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 299224us
I20260806 20:34:57.870448 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:57.871668 14552 raft_consensus.cc:3032] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:57.872076 14552 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:57.873201 14552 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:57.874569 14552 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:57.886459 14549 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:57.889127 14633 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:34:57.891680 14550 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:34:57.882735 15748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.006s sys 0.017s
I20260806 20:34:57.863500 15739 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260806 20:34:57.894043 15740 raft_consensus.cc:686] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:34:57.979318 14476 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:57.948642 15709 raft_consensus.cc:979] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: : Instructing follower 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:34:58.001034 14552 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:57.915786 14548 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:34:58.008657 15753 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260806 20:34:58.013345 15186 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 LEARNER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:34:58.016342 15724 raft_consensus.cc:1067] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Signalling peer 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:34:58.020352 15184 raft_consensus.cc:2932] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:58.027473 15724 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:34:58.048153 14479 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:58.050216 14552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:54498
W20260806 20:34:57.910774 15748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.006s sys 0.017s
I20260806 20:34:58.067551 14369 catalog_manager.cc:5495] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 2, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:34:58.070950 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:34:58.090631 14628 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:34:58.091722 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 64649.7us
I20260806 20:34:58.052727 15661 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:34:58.093674 15739 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 20:34:58.104099 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 98389.8us
I20260806 20:34:58.064271 14370 catalog_manager.cc:5495] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 3 to 4, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:34:58.126847 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 101684us
I20260806 20:34:58.164069 15735 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:34:58.138777 15740 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:58.147642 15766 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260806 20:34:58.111176 15765 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.132s
W20260806 20:34:58.346829 14625 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.060s sys 0.059s
W20260806 20:34:58.357036 14625 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.060s sys 0.059s
I20260806 20:34:58.402453 15726 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.210s
W20260806 20:34:58.359325 14547 consensus_meta.cc:287] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.592s user 0.013s sys 0.005s
I20260806 20:34:58.165246 15735 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: NON_VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: true } }
I20260806 20:34:58.613647 15735 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:34:58.616501 15761 log.cc:924] Time spent T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Append to log took a long time: real 0.270s user 0.001s sys 0.003s
W20260806 20:34:58.720484 14476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.028s sys 0.037s
W20260806 20:34:58.720903 14476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s user 0.028s sys 0.037s
W20260806 20:34:58.728372 14471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.011s sys 0.063s
W20260806 20:34:58.728950 14471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s user 0.011s sys 0.063s
W20260806 20:34:58.775077 14477 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.022s sys 0.049s
W20260806 20:34:58.775599 14477 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.022s sys 0.050s
I20260806 20:34:58.843310 14555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:58.844079 14555 raft_consensus.cc:595] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
W20260806 20:34:58.860884 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.001s sys 0.016s
W20260806 20:34:58.859769 14628 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s user 0.013s sys 0.018s
W20260806 20:34:58.861428 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.002s sys 0.016s
W20260806 20:34:58.863862 14628 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s user 0.013s sys 0.018s
W20260806 20:34:58.873524 15184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.014s sys 0.018s
W20260806 20:34:58.876145 15184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.014s sys 0.018s
W20260806 20:34:58.887708 14549 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.015s sys 0.017s
W20260806 20:34:58.888149 14549 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s user 0.015s sys 0.017s
W20260806 20:34:58.885658 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.015s sys 0.004s
W20260806 20:34:58.898226 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.015s sys 0.004s
I20260806 20:34:58.913364 14553 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:58.914144 14553 raft_consensus.cc:595] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:58.914311 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:58.924125 14625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:34:58.925072 14625 raft_consensus.cc:595] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:58.930541 14554 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:34:58.931393 14554 raft_consensus.cc:595] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
W20260806 20:34:58.934820 15771 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.074s user 0.002s sys 0.000s
I20260806 20:34:58.938066 14474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:34:58.938838 14474 raft_consensus.cc:595] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:34:58.944777 14474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:34:58.945533 14474 raft_consensus.cc:595] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:34:58.968010 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 20:34:58.970444 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:34:58.971412 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:34:58.968154 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.035s sys 0.037s
W20260806 20:34:58.980901 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s user 0.035s sys 0.037s
W20260806 20:34:58.987061 14479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.015s sys 0.029s
W20260806 20:34:58.987629 14479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.884s user 0.015s sys 0.029s
W20260806 20:34:58.997387 14551 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.009s sys 0.031s
W20260806 20:34:58.997781 14551 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s user 0.009s sys 0.031s
I20260806 20:34:58.964545 15664 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:34:58.968642 14548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.027s sys 0.024s
I20260806 20:34:59.048157 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:34:59.052058 14548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.028s sys 0.024s
I20260806 20:34:59.052855 14476 raft_consensus.cc:2932] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:34:59.038772 15664 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:34:59.150211 14547 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.009s sys 0.066s
W20260806 20:34:59.185777 14547 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.009s sys 0.067s
I20260806 20:34:59.187409 14552 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:59.187920 14552 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:59.187170 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1016) took 1448 ms. Trace:
I20260806 20:34:59.204136 14552 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:34:59.215754 15776 log.cc:924] Time spent T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260806 20:34:59.224107 15761 log.cc:924] Time spent T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
W20260806 20:34:59.227355 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 265553us
I20260806 20:34:59.339200 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 41) took 1233 ms. Trace:
W20260806 20:34:59.330607 15769 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 20:34:59.348611 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 43) took 1250 ms. Trace:
W20260806 20:34:59.373250 15777 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20260806 20:34:59.373595 15775 log.cc:924] Time spent T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20260806 20:34:59.374693 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1027) took 1343 ms. Trace:
W20260806 20:34:59.375746 15780 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260806 20:34:59.375128 14551 rpcz_store.cc:276] 0806 20:34:58.031424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.031611 (+ 187us) service_pool.cc:227] Handling call
0806 20:34:58.032063 (+ 452us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:34:58.141713 (+109650us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:34:58.141760 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:34:58.148831 (+ 7071us) log.cc:842] Serialized 26 byte log entry
0806 20:34:58.998164 (+849333us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:34:58.998202 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:58.998282 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:34:59.374500 (+376218us) raft_consensus.cc:1671] finished
0806 20:34:59.374520 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.374640 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":124132,"raft.queue_time_us":109261,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":16051,"thread_start_us":763934,"threads_started":1,"wal-append.queue_time_us":1016634}
I20260806 20:34:59.407603 14625 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:34:59.339692 14628 rpcz_store.cc:276] 0806 20:34:58.099074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.104694 (+ 5620us) service_pool.cc:227] Handling call
0806 20:34:58.105028 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:34:58.124555 (+ 19527us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:34:58.124598 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:34:58.125157 (+ 559us) log.cc:842] Serialized 26 byte log entry
0806 20:34:58.868787 (+743630us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:34:58.868837 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:58.868931 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:34:59.332546 (+463615us) raft_consensus.cc:1671] finished
0806 20:34:59.332572 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.332724 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19107,"raft.queue_time_us":37151,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":14186,"thread_start_us":743276,"threads_started":1,"wal-append.queue_time_us":916386}
I20260806 20:34:59.349159 14479 rpcz_store.cc:276] 0806 20:34:58.094637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.102458 (+ 7821us) service_pool.cc:227] Handling call
0806 20:34:58.102779 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:34:58.103100 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:34:58.103125 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:34:58.103551 (+ 426us) log.cc:842] Serialized 26 byte log entry
0806 20:34:58.988195 (+884644us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:34:58.988245 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:58.988332 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:34:59.344650 (+356318us) raft_consensus.cc:1671] finished
0806 20:34:59.344677 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.344834 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9616,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":10435,"thread_start_us":884247,"threads_started":1,"wal-append.queue_time_us":1114762}
I20260806 20:34:59.461661 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:59.472020 15664 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:34:59.484779 15784 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.163s user 0.001s sys 0.001s
W20260806 20:34:59.489516 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 202289us
I20260806 20:34:59.481880 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 58) took 1104 ms. Trace:
I20260806 20:34:59.201426 14547 rpcz_store.cc:276] 0806 20:34:57.738616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:57.740817 (+ 2201us) service_pool.cc:227] Handling call
0806 20:34:57.763604 (+ 22787us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:34:57.763970 (+ 366us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:34:58.419842 (+655872us) log.cc:842] Serialized 14 byte log entry
0806 20:34:59.186466 (+766624us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:34:59.186593 (+ 127us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:34:59.186674 (+ 81us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:59.186827 (+ 153us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.187027 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":611,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":158,"thread_start_us":628678,"threads_started":1,"wal-append.queue_time_us":856448}
W20260806 20:34:59.507807 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 196523us
I20260806 20:34:59.510483 14551 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:34:59.513139 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 285990us
I20260806 20:34:59.524803 15730 raft_consensus.cc:979] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:59.526497 15730 raft_consensus.cc:1067] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:59.527272 15730 raft_consensus.cc:979] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:59.527477 14552 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:34:59.527681 15730 raft_consensus.cc:979] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:59.528043 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:54498 (request call id 1026) took 1502 ms. Trace:
I20260806 20:34:59.528131 15730 raft_consensus.cc:1067] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
W20260806 20:34:59.528481 15782 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20260806 20:34:59.529212 15759 raft_consensus.cc:1067] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:34:59.529881 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1302) took 1422 ms. Trace:
I20260806 20:34:59.514079 15740 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260806 20:34:59.530894 14552 rpcz_store.cc:276] 0806 20:34:58.025197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.050087 (+ 24890us) service_pool.cc:227] Handling call
0806 20:34:59.525858 (+1475771us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400139cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34eada7 00007fb9b8436ee0 00007fb9b1b28784 00007fb9b1b28712 00007fb9b1b28641 00007fb9b1b285bc 00007fb9b1b27872 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:34:59.527914 (+ 2056us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2613586816}
W20260806 20:34:59.561218 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47469.9us
I20260806 20:34:59.565122 14595 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:34:59.609740 15184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:59.610520 15184 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:59.610929 15184 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:59.622082 15187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:59.622857 15187 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:59.623266 15187 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:34:59.625167 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 52130us
W20260806 20:34:59.636533 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 105060us
I20260806 20:34:59.647442 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:34:59.648126 14628 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:59.651551 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 59) took 1273 ms. Trace:
I20260806 20:34:59.657788 14549 rpcz_store.cc:276] 0806 20:34:58.378421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.378564 (+ 143us) service_pool.cc:227] Handling call
0806 20:34:58.378939 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:34:58.382902 (+ 3963us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:34:58.382932 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:34:58.383565 (+ 633us) log.cc:842] Serialized 26 byte log entry
0806 20:34:58.888582 (+505017us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:34:58.888636 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:58.888728 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:34:59.651349 (+762621us) raft_consensus.cc:1671] finished
0806 20:34:59.651372 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.651495 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23305,"raft.queue_time_us":4910,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":10538,"thread_start_us":504432,"threads_started":1,"wal-append.queue_time_us":946864}
I20260806 20:34:59.636714 15728 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:34:59.617049 15739 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260806 20:34:59.638902 15716 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:34:59.699318 15779 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:34:59.743531 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 144469us
W20260806 20:34:59.720968 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 209160us
I20260806 20:34:59.660321 15799 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:34:59.750366 15184 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.759019 15184 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:34:59.798302 15187 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.800330 14628 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
W20260806 20:34:59.802477 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 43636.2us
I20260806 20:34:59.721074 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:34:59.805652 15183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:60780
I20260806 20:34:59.806406 15183 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:34:59.806784 15183 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:59.714273 15728 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.809018 15185 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:59.504886 14548 rpcz_store.cc:276] 0806 20:34:58.378011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.389449 (+ 11438us) service_pool.cc:227] Handling call
0806 20:34:58.394552 (+ 5103us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:34:58.408029 (+ 13477us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:34:58.408095 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:34:58.408886 (+ 791us) log.cc:842] Serialized 26 byte log entry
0806 20:34:59.210916 (+802030us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:34:59.225822 (+ 14906us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:59.247060 (+ 21238us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:34:59.455130 (+208070us) raft_consensus.cc:1671] finished
0806 20:34:59.457996 (+ 2866us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.478516 (+ 20520us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8405,"raft.queue_time_us":18495,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":35544,"thread_start_us":736449,"threads_started":1,"wal-append.queue_time_us":807331}
W20260806 20:34:59.809211 15783 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.434s user 0.009s sys 0.000s
I20260806 20:34:59.825963 15187 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:34:59.805083 15776 log.cc:924] Time spent T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.059s user 0.000s sys 0.008s
I20260806 20:34:59.722297 15716 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.821594 15743 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:34:59.853870 15728 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:34:59.548771 14631 rpcz_store.cc:276] 0806 20:34:58.107473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.145652 (+ 38179us) service_pool.cc:227] Handling call
0806 20:34:58.164331 (+ 18679us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:34:58.566260 (+401929us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:34:58.566309 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:34:58.593250 (+ 26941us) log.cc:842] Serialized 572 byte log entry
0806 20:34:59.118182 (+524932us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:34:59.124714 (+ 6532us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:59.186785 (+ 62071us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54002404d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:34:59.186862 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:34:59.523540 (+336678us) raft_consensus.cc:1671] finished
0806 20:34:59.529502 (+ 5962us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.529721 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":922908928,"thread_start_us":453461,"threads_started":1,"wal-append.queue_time_us":634942}
W20260806 20:34:59.860483 15805 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
W20260806 20:34:59.864764 15795 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20260806 20:34:59.866719 15809 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:34:59.866976 15665 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:34:59.877345 15800 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:34:59.878621 15716 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:34:59.878396 15800 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:59.871649 15743 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.884166 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:59.886494 15665 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:34:59.857956 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:34:59.889556 14547 raft_consensus.cc:3032] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:34:59.892997 15722 raft_consensus.cc:979] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: : Instructing follower 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:59.894695 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:34:59.895463 14471 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:59.890103 14547 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:34:59.910953 15183 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.917320 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:59.918041 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:34:59.900141 15665 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:59.893693 15722 raft_consensus.cc:1067] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Signalling peer 92828dd6dbe74f29a04d4e702b502621 to start an election
I20260806 20:34:59.826975 15799 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.894914 15809 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:34:59.925803 15743 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:34:59.933116 15185 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:34:59.950690 14471 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:34:59.943017 14547 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:34:59.971738 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 321477us
I20260806 20:34:59.947403 15800 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:34:59.963269 15799 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:34:59.974714 14547 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:34:59.980166 15809 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:34:59.980887 15785 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.472s user 0.000s sys 0.003s
I20260806 20:34:59.982214 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60780 (request call id 251) took 1429 ms. Trace:
I20260806 20:34:59.982661 15186 rpcz_store.cc:276] 0806 20:34:58.553257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:58.554756 (+ 1499us) service_pool.cc:227] Handling call
0806 20:34:58.570022 (+ 15266us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:34:58.570397 (+ 375us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:34:58.570420 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:34:58.571589 (+ 1169us) log.cc:842] Serialized 572 byte log entry
0806 20:34:59.057105 (+485516us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:34:59.057164 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:34:59.057260 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:34:59.929239 (+871979us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:34:59.981791 (+ 52552us) raft_consensus.cc:1671] finished
0806 20:34:59.981823 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:34:59.982115 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87532928,"thread_start_us":485134,"threads_started":1,"wal-append.queue_time_us":836485}
I20260806 20:34:59.977052 15183 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.018718 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.054332 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.056819 14471 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:00.061468 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.059106 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.065518 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:00.082278 14551 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.009s sys 0.016s
W20260806 20:35:00.082758 14551 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.538s user 0.009s sys 0.016s
W20260806 20:35:00.085872 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 447819us
W20260806 20:35:00.086755 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 262201us
I20260806 20:35:00.093567 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.094347 14625 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:00.096738 15628 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:00.097335 15628 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.099283 15628 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:00.100912 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:00.101725 14553 raft_consensus.cc:3032] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:00.102056 14553 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:00.102820 14553 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.104954 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:00.108412 14521 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:00.115260 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:00.116169 14548 raft_consensus.cc:3032] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:00.116536 14548 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:00.128656 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.012s sys 0.020s
W20260806 20:35:00.132790 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.012s sys 0.020s
I20260806 20:35:00.132721 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:00.135741 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 570698us
I20260806 20:35:00.135996 15187 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:00.124313 15766 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.006s sys 0.005s
I20260806 20:35:00.137132 14518 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:00.146332 15643 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:00.146896 15643 raft_consensus.cc:686] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:00.147913 15643 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:00.142859 15766 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.582s user 0.006s sys 0.006s
W20260806 20:35:00.129650 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 323368us
W20260806 20:35:00.154706 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 246339us
I20260806 20:35:00.165752 14444 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:00.177932 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:00.178536 14626 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:00.180891 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.183307 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:00.189843 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:00.190511 14473 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:00.189903 15150 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:00.194029 15788 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.015s sys 0.035s
W20260806 20:35:00.194392 15788 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.820s user 0.016s sys 0.035s
I20260806 20:35:00.194943 15818 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:00.195685 15818 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.198726 15818 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:00.201062 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:00.203011 14444 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:00.207019 15755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.004s sys 0.007s
I20260806 20:35:00.206784 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:00.207393 15755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.735s user 0.004s sys 0.007s
I20260806 20:35:00.208952 15153 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:00.209128 15185 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:00.219533 14547 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:00.220223 15739 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:00.139089 14548 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.223577 14626 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:00.225818 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:00.226512 14474 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:00.229085 15765 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:00.229239 15153 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:00.229651 15765 raft_consensus.cc:686] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:00.316789 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 113875us
I20260806 20:35:00.317865 14444 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:00.324772 14548 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:00.227317 14473 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
W20260806 20:35:00.328724 15599 consensus_queue.cc:1159] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "148bf2a11dba423ab364371f0458e871" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:35:00.329934 15633 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:00.334018 15754 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.504s user 0.007s sys 0.028s
I20260806 20:35:00.334863 15819 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:00.336405 14470 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:54500
I20260806 20:35:00.337167 14470 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:35:00.337528 14470 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:00.339752 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 106672us
I20260806 20:35:00.340188 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:00.230611 15765 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.369755 15793 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:00.369624 15807 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:00.341779 15691 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:00.370462 15807 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:00.692826 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 375011us
I20260806 20:35:00.693315 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:00.410071 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:00.568104 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:00.436084 14470 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.388413 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:00.666946 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 295057us
I20260806 20:35:00.713269 15186 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:00.718869 15727 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.404839 15797 raft_consensus.cc:979] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: : Instructing follower 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:35:00.723764 14626 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 20:35:00.564011 15819 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.724550 15794 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.724622 15740 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.531852 14548 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:00.638811 15735 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.399852 15793 raft_consensus.cc:686] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:00.552772 15746 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.655405 15820 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.390410 15821 raft_consensus.cc:979] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:35:00.731034 14470 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.715247 15806 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:00.665340 15804 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.687901 15691 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.719362 15727 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.739533 15833 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.744129 15743 raft_consensus.cc:1067] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:35:00.743652 15796 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.738396 15746 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.734300 15740 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.734656 15794 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.771205 15794 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.740427 15735 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.750851 15739 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.747231 15691 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:00.783161 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 52130.5us
I20260806 20:35:00.783525 15847 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.784235 15847 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.784881 15691 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.758338 15185 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 20:35:00.752497 15665 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.753288 15779 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.804222 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:00.805084 14628 raft_consensus.cc:3032] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:00.805461 14628 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:00.808827 14628 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.792371 15807 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.734823 15819 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.742451 15820 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.799263 14475 raft_consensus.cc:3032] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:00.815274 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.792610 15665 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.816504 14475 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:00.758117 15833 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.787962 15739 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.794004 15779 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:00.811277 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.830219 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:00.834780 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 60252.7us
I20260806 20:35:00.832118 14471 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:00.749838 15804 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:00.840663 15838 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260806 20:35:00.791951 15793 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.842267 14475 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.846750 15847 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:00.773082 15796 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:00.849617 15665 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.850015 15819 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.853351 15151 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:00.851572 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.836665 15727 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.851964 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:00.863200 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 68405.1us
I20260806 20:35:00.868753 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:00.851272 15844 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:00.871268 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:00.863185 15830 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:00.868362 15740 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:00.875465 15844 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.040321 14369 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45942 (request call id 213) took 3279 ms. Trace:
I20260806 20:35:01.112064 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:00.930371 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.101341 14370 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45970 (request call id 230) took 3552 ms. Trace:
I20260806 20:35:01.082581 15755 raft_consensus.cc:2932] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:01.006152 15794 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.114672 15794 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:01.115198 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.116127 14548 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:01.044775 15806 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:00.894279 15837 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:01.117131 15806 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:01.024648 15835 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:00.987066 14368 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56582 (request call id 137) took 3746 ms. Trace:
I20260806 20:35:00.878180 15746 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:01.041952 14628 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:01.059039 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 310817us
I20260806 20:35:01.066525 14553 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:00.936359 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:01.168727 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.170128 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.170552 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.171456 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.113605 14370 rpcz_store.cc:276] 0806 20:34:57.548557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:57.548732 (+ 175us) service_pool.cc:227] Handling call
0806 20:34:58.060004 (+511272us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400002dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b944dd6c 00007fb9b9547c3e 00007fb9b7859fb4 00007fb9b7859f42 00007fb9b7859e71 00007fb9b7859dec 00007fb9b78590a2 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:01.101235 (+3041231us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:34:58.373215 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:34:58.374401 (+ 1186us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:34:58.374457 (+ 56us) write_op.cc:435] Acquired schema lock
0806 20:34:58.374513 (+ 56us) tablet.cc:660] Decoding operations
0806 20:34:58.511607 (+137094us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:34:58.550957 (+ 39350us) write_op.cc:641] Partition lock acquired for write op
0806 20:34:58.557282 (+ 6325us) tablet.cc:683] Acquiring locks for 1 operations
0806 20:34:58.589967 (+ 32685us) tablet.cc:699] Row locks acquired
0806 20:34:58.590055 (+ 88us) write_op.cc:260] PREPARE: finished
0806 20:34:58.602731 (+ 12676us) write_op.cc:270] Start()
0806 20:34:58.625292 (+ 22561us) write_op.cc:276] Timestamp: P: 1786048498601232 usec, L: 0
0806 20:34:58.625329 (+ 37us) op_driver.cc:331] REPLICATION: starting
0806 20:34:58.626300 (+ 971us) log.cc:842] Serialized 603 byte log entry
0806 20:35:00.713943 (+2087643us) op_driver.cc:444] REPLICATION: finished
0806 20:35:00.750045 (+ 36102us) write_op.cc:301] APPLY: starting
0806 20:35:00.752891 (+ 2846us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:00.774407 (+ 21516us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:00.793531 (+ 19124us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:00.804233 (+ 10702us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:00.812658 (+ 8425us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 20:35:00.816703 (+ 4045us) write_op.cc:312] APPLY: finished
0806 20:35:00.829428 (+ 12725us) log.cc:842] Serialized 23 byte log entry
0806 20:35:00.837956 (+ 8528us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:00.839072 (+ 1116us) write_op.cc:454] Released schema lock
0806 20:35:00.844697 (+ 5625us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:00.858653 (+ 13956us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:00.861571 (+ 2918us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":186804,"reported_tablets":1,"spinlock_wait_cycles":1171320192,"child_traces":[["op",{"apply.queue_time_us":26820,"apply.run_cpu_time_us":2937,"apply.run_wall_time_us":121357,"num_ops":1,"prepare.queue_time_us":260830,"prepare.run_cpu_time_us":5033,"prepare.run_wall_time_us":253468,"replication_time_us":1850618}]]}
I20260806 20:35:01.078570 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 69) took 1254 ms. Trace:
I20260806 20:35:01.171425 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:01.180897 14551 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:01.179875 14549 rpcz_store.cc:276] 0806 20:34:59.824022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:59.824173 (+ 151us) service_pool.cc:227] Handling call
0806 20:34:59.824664 (+ 491us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:00.086295 (+261631us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:00.086325 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:00.130679 (+ 44354us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b2c00000238. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:00.137869 (+ 7190us) log.cc:842] Serialized 26 byte log entry
0806 20:35:00.428593 (+290724us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:00.428630 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:01.078247 (+649617us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b540018ddd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:01.078293 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:01.078341 (+ 48us) raft_consensus.cc:1671] finished
0806 20:35:01.078355 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:01.078508 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2923,"raft.queue_time_us":30049,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":9751,"spinlock_wait_cycles":108998144,"thread_start_us":502915,"threads_started":2,"wal-append.queue_time_us":589954}
I20260806 20:35:00.880393 15735 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:01.184010 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 77102.6us
I20260806 20:35:01.063980 15836 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:01.190902 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:00.888932 15830 raft_consensus.cc:686] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:01.024729 15740 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.045603 15796 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.198442 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.198185 14629 raft_consensus.cc:2932] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:35:01.201399 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 391330us
I20260806 20:35:01.073832 15833 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.046572 15739 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.112564 14369 rpcz_store.cc:276] 0806 20:34:57.760860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:57.761009 (+ 149us) service_pool.cc:227] Handling call
0806 20:34:58.065070 (+304061us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b5400002dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b944dd6c 00007fb9b9547c3e 00007fb9b7859fb4 00007fb9b7859f42 00007fb9b7859e71 00007fb9b7859dec 00007fb9b78590a2 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:01.040239 (+2975169us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:34:58.134559 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:34:58.147785 (+ 13226us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:34:58.147863 (+ 78us) write_op.cc:435] Acquired schema lock
0806 20:34:58.147956 (+ 93us) tablet.cc:660] Decoding operations
0806 20:34:58.248159 (+100203us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:34:58.259141 (+ 10982us) write_op.cc:641] Partition lock acquired for write op
0806 20:34:58.265232 (+ 6091us) tablet.cc:683] Acquiring locks for 1 operations
0806 20:34:58.343692 (+ 78460us) tablet.cc:699] Row locks acquired
0806 20:34:58.350280 (+ 6588us) write_op.cc:260] PREPARE: finished
0806 20:34:58.370827 (+ 20547us) write_op.cc:270] Start()
0806 20:34:58.371076 (+ 249us) write_op.cc:276] Timestamp: P: 1786048498370750 usec, L: 0
0806 20:34:58.371098 (+ 22us) op_driver.cc:331] REPLICATION: starting
0806 20:34:58.372410 (+ 1312us) log.cc:842] Serialized 544 byte log entry
0806 20:35:00.195006 (+1822596us) op_driver.cc:444] REPLICATION: finished
0806 20:35:00.621424 (+426418us) write_op.cc:301] APPLY: starting
0806 20:35:00.631869 (+ 10445us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:00.638263 (+ 6394us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:00.638571 (+ 308us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:00.655024 (+ 16453us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:00.711640 (+ 56616us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 20:35:00.720836 (+ 9196us) write_op.cc:312] APPLY: finished
0806 20:35:00.733795 (+ 12959us) log.cc:842] Serialized 23 byte log entry
0806 20:35:00.741776 (+ 7981us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:00.742361 (+ 585us) write_op.cc:454] Released schema lock
0806 20:35:00.744678 (+ 2317us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:00.751644 (+ 6966us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:00.754543 (+ 2899us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":240227,"reported_tablets":1,"spinlock_wait_cycles":694458752,"child_traces":[["op",{"apply.queue_time_us":370069,"apply.run_cpu_time_us":2875,"apply.run_wall_time_us":151535,"num_ops":1,"prepare.queue_time_us":24464,"prepare.run_cpu_time_us":7103,"prepare.run_wall_time_us":248753,"replication_time_us":1823781,"thread_start_us":231055,"threads_started":1}]]}
I20260806 20:35:01.248907 14549 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:01.251688 14470 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:01.142416 15846 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:01.264968 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.117707 15794 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.258505 14628 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:01.253935 15740 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:01.164709 14368 rpcz_store.cc:276] 0806 20:34:57.236970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:34:57.237113 (+ 143us) service_pool.cc:227] Handling call
0806 20:35:00.983543 (+3746430us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:34:57.296792 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:34:57.297543 (+ 751us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:34:57.297585 (+ 42us) write_op.cc:435] Acquired schema lock
0806 20:34:57.297630 (+ 45us) tablet.cc:660] Decoding operations
0806 20:34:57.298222 (+ 592us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:34:57.298350 (+ 128us) write_op.cc:641] Partition lock acquired for write op
0806 20:34:57.298391 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0806 20:34:57.298631 (+ 240us) tablet.cc:699] Row locks acquired
0806 20:34:57.298659 (+ 28us) write_op.cc:260] PREPARE: finished
0806 20:34:57.298921 (+ 262us) write_op.cc:270] Start()
0806 20:34:57.299096 (+ 175us) write_op.cc:276] Timestamp: P: 1786048497298850 usec, L: 0
0806 20:34:57.299142 (+ 46us) op_driver.cc:331] REPLICATION: starting
0806 20:34:57.300223 (+ 1081us) log.cc:842] Serialized 539 byte log entry
0806 20:34:57.617679 (+317456us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b05ad5fe 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb
0806 20:34:59.370005 (+1752326us) op_driver.cc:444] REPLICATION: finished
0806 20:34:59.881873 (+511868us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b05ad5fe 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b6f03410
0806 20:35:00.360830 (+478957us) write_op.cc:301] APPLY: starting
0806 20:35:00.360920 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:00.361056 (+ 136us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:00.361076 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:00.361363 (+ 287us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:00.361520 (+ 157us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 20:35:00.361571 (+ 51us) write_op.cc:312] APPLY: finished
0806 20:35:00.374462 (+ 12891us) log.cc:842] Serialized 23 byte log entry
0806 20:35:00.395873 (+ 21411us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:00.401525 (+ 5652us) write_op.cc:454] Released schema lock
0806 20:35:00.427110 (+ 25585us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:00.464701 (+ 37591us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:00.468123 (+ 3422us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":986347,"apply.run_cpu_time_us":2828,"apply.run_wall_time_us":131707,"num_ops":1,"prepare.queue_time_us":52326,"prepare.run_cpu_time_us":35408,"prepare.run_wall_time_us":758669,"raft.queue_time_us":2572654,"raft.run_cpu_time_us":88070,"raft.run_wall_time_us":1358487,"replication_time_us":2054433,"spinlock_wait_cycles":15606272,"thread_start_us":2022031,"threads_started":4,"wal-append.queue_time_us":916742}]]}
W20260806 20:35:01.260393 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 56135.8us
W20260806 20:35:01.249930 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 52332.6us
I20260806 20:35:01.269776 14441 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:01.270875 14441 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.271864 14441 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.165089 15835 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:01.189673 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 117650us
I20260806 20:35:01.255434 14518 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:01.276821 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.277462 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.286257 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.265461 15846 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.117396 15837 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.261338 15726 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.294706 14628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17eec21790894c559ad9623765626416"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:35:01.295526 14628 raft_consensus.cc:595] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:01.295933 15151 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:01.275123 15692 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:01.313306 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 124452us
I20260806 20:35:01.278528 15821 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:01.317535 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.200802 15836 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:01.337720 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.344280 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.294095 15183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:59384
I20260806 20:35:01.302155 15842 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.046325 15853 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.467s
I20260806 20:35:01.282135 15856 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:01.368683 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:01.318173 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 63200.6us
I20260806 20:35:01.369395 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.374356 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.365212 14551 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:35:01.380367 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 119381us
I20260806 20:35:01.569602 15153 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.521031 14553 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:01.534030 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.577744 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:01.579135 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 304078us
W20260806 20:35:01.583361 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 266187us
I20260806 20:35:01.317086 15863 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 20:35:01.580632 14553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 72) took 1473 ms (client timeout 1710 ms). Trace:
I20260806 20:35:01.570703 15183 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:35:01.374966 15710 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:01.319553 15846 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:01.588096 14553 rpcz_store.cc:269] 0806 20:35:00.098695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:00.100656 (+ 1961us) service_pool.cc:227] Handling call
0806 20:35:00.335414 (+234758us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b1800014ac0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b352e856 00007fb9b3480f7e 00007fb9b34ea367 00007fb9b34ebb4e 00007fb9b34f7228 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c
0806 20:35:01.066221 (+730807us) spinlock_profiling.cc:243] Waited 730 ms on lock 0x7b58002df8b0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b346ef63 00007fb9b34d2aba 00007fb9b34ea3af 00007fb9b34ebb4e 00007fb9b34f7228 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c
0806 20:35:01.572393 (+506172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1962503808}
I20260806 20:35:01.570616 15835 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.321763 15837 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.591604 14442 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.488655 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.610538 15665 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:01.355136 15830 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:01.613054 15187 raft_consensus.cc:1226] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:01.614650 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:01.614900 14442 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.316385 15726 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.574348 15811 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:01.521031 15844 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.447322 15815 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:01.384305 15821 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.617914 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.623651 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.615705 15807 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:01.617486 14442 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.618728 14478 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:01.618824 14551 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:01.620935 15186 raft_consensus.cc:2932] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:01.620988 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.578567 15823 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:01.586961 15863 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.592485 15710 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:01.590113 15740 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.629403 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.594470 15183 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:01.630647 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.631098 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.624390 14630 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:01.643836 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.614187 14474 raft_consensus.cc:2932] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:01.627522 15821 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:01.659258 14518 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:01.659833 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 66416.6us
I20260806 20:35:01.660765 15692 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:01.661377 14442 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:01.661852 14521 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:01.706013 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 44263us
I20260806 20:35:01.706120 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.662497 15807 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.706794 15816 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:01.702943 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.636389 15844 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:01.701849 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.638350 15863 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:01.643802 15710 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:01.657464 15665 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:01.617823 15804 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:01.710453 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:01.718158 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:01.715006 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 53742us
I20260806 20:35:01.723846 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:01.705227 15643 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:01.658378 15727 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260806 20:35:01.707026 15807 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.628778 15823 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.720067 15818 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:01.839229 15872 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:01.840564 15872 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:01.659330 15150 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:01.716310 15844 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.733017 15692 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:01.629654 15815 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:01.591046 15842 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.930850 15867 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:01.855978 15818 raft_consensus.cc:686] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:01.736987 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:01.847929 15872 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:01.740693 15836 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:01.985112 15815 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:02.014719 15870 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.271s user 0.001s sys 0.000s
W20260806 20:35:01.975193 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 341591us
I20260806 20:35:01.920691 15739 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:01.982173 15856 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:01.710182 15816 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:01.934717 14368 catalog_manager.cc:5495] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 2 to 3, term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to <none>, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:01.926957 15796 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:02.011133 15865 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.328s
I20260806 20:35:02.010588 15850 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:01.931751 15867 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.047215 14368 catalog_manager.cc:5495] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:01.992206 15807 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.053783 14368 catalog_manager.cc:5495] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:02.021036 15599 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:02.063105 14368 catalog_manager.cc:5495] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:01.989815 15726 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.045064 15872 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:02.069696 15850 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:02.124123 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.075393 15847 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:01.922310 15692 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.048377 15739 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.137209 14551 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:35:02.138762 15871 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20260806 20:35:02.139539 15843 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:02.139627 15867 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.140311 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.071699 15599 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.140936 14628 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:02.127722 15850 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:02.142805 15755 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.028s user 0.014s sys 0.000s
I20260806 20:35:02.159879 14521 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:02.117340 15836 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260806 20:35:02.114439 15872 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.136521 15664 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:02.168893 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.135334 15844 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.124516 15153 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:02.175347 15878 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:02.176010 15878 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:02.176512 15878 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:02.175132 15796 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:02.158653 15820 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:02.069842 15856 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.130630 15150 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:02.170193 15806 consensus_meta.cc:287] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.052s user 0.013s sys 0.003s
I20260806 20:35:02.140120 15843 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.171494 15823 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.135880 15847 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.198468 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.198366 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.190501 15865 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:02.154937 15818 consensus_queue.cc:237] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.170303 15664 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.194308 15835 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:02.219758 15628 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:02.197816 15727 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:02.220329 15628 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.227487 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:02.228412 15184 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
W20260806 20:35:02.197180 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 83693.4us
I20260806 20:35:02.217669 15842 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.232985 15153 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:02.234364 15628 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.234375 15664 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:02.208986 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 87467.2us
I20260806 20:35:02.256640 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.193537 15806 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.217381 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:02.245215 15183 consensus_meta.cc:287] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.575s user 0.006s sys 0.006s
I20260806 20:35:02.239840 15865 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:02.285621 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.296785 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.244302 15746 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:02.297554 14632 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:02.312011 14521 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:02.313858 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1363) took 1123 ms. Trace:
I20260806 20:35:02.237227 15727 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.307983 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.314332 14626 rpcz_store.cc:276] 0806 20:35:01.190475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.190574 (+ 99us) service_pool.cc:227] Handling call
0806 20:35:01.190822 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:01.318619 (+127797us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:01.318652 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:01.318698 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:01.318731 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:02.313633 (+994902us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540006f9d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:02.313676 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:02.313801 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293447424}
I20260806 20:35:02.281037 15692 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.371055 15883 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:02.302712 15739 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:02.373927 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.370718 15859 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:02.385530 14553 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:35:02.390161 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 687710us
I20260806 20:35:02.390686 15727 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.391746 15880 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:02.379963 15150 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:02.393918 15865 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:02.394819 15150 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:02.398699 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:02.399537 15184 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:02.276757 15779 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:02.319362 15599 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.291298 15816 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.404958 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:02.406446 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:02.407274 15843 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.300494 15183 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.314114 15746 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:02.415323 15876 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.203s user 0.001s sys 0.000s
W20260806 20:35:02.410295 14518 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.905s (SENT) [suppressed 1 similar messages]
I20260806 20:35:02.383104 15872 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.393064 15880 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.402410 15859 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:02.432771 15874 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20260806 20:35:02.435074 15778 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:02.375236 15883 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.413240 15865 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:02.416503 15856 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.450755 15183 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.451097 15185 raft_consensus.cc:1226] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:02.463186 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:59384 (request call id 310) took 1184 ms. Trace:
I20260806 20:35:02.463786 15183 rpcz_store.cc:276] 0806 20:35:01.279250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.283275 (+ 4025us) service_pool.cc:227] Handling call
0806 20:35:02.463034 (+1179759us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:02.465592 15710 consensus_meta.cc:287] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.742s user 0.008s sys 0.005s
I20260806 20:35:02.466171 15710 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.467777 14550 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:02.468858 14548 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:02.456008 15665 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:02.471233 15665 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:02.479889 15806 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.454937 15880 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.437664 15859 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:02.501747 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 243727us
W20260806 20:35:02.495000 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 263471us
I20260806 20:35:02.447283 15872 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:02.520390 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:02.522637 14441 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:02.525931 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:02.526535 14628 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:02.527941 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:02.530099 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:02.549552 14477 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:02.555585 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:02.556289 14477 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:02.557116 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:02.567399 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:02.570354 15883 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.567655 15710 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.572191 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:02.492412 15847 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.573510 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:02.596853 14477 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:02.435923 15778 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.553248 15754 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:02.577495 14628 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:02.574357 15865 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:02.539901 15872 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.612994 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:02.613686 14470 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
W20260806 20:35:02.612888 15821 consensus_meta.cc:287] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.909s user 0.011s sys 0.007s
I20260806 20:35:02.616258 15821 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.618600 15746 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:02.605083 15754 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:02.558651 15811 consensus_queue.cc:1046] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
W20260806 20:35:02.631740 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 356230us
I20260806 20:35:02.620898 15754 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:02.629287 15863 consensus_meta.cc:287] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.905s user 0.006s sys 0.011s
I20260806 20:35:02.620339 15860 consensus_queue.cc:1046] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.092s
I20260806 20:35:02.627718 15872 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:02.621855 15153 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:02.638162 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:02.687872 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.683377 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.638360 15821 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:02.625717 15878 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.666145 15778 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:02.699179 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.701435 15665 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.704397 15878 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.705581 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:02.706216 14630 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:02.682490 15863 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.707716 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.710748 15863 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.711445 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:02.621109 15865 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:02.715004 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:02.713675 15665 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:02.717396 14630 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:02.717797 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:02.717069 14474 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.842s user 0.007s sys 0.010s
I20260806 20:35:02.718449 14521 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:02.694955 15183 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:02.720685 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 88114.3us
I20260806 20:35:02.722716 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1275) took 1128 ms. Trace:
I20260806 20:35:02.723059 14471 rpcz_store.cc:276] 0806 20:35:01.594122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.598070 (+ 3948us) service_pool.cc:227] Handling call
0806 20:35:01.609389 (+ 11319us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:02.000596 (+391207us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:02.000643 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:02.000783 (+ 140us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:02.000830 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:02.722483 (+721653us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b5400333bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:02.722526 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:02.722665 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":898915712}
I20260806 20:35:02.725371 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:02.726804 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.728366 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.728323 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.731940 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.736390 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:02.741827 14551 raft_consensus.cc:2379] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:02.829389 15863 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:02.729985 15844 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
W20260806 20:35:02.826376 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 216649us
I20260806 20:35:02.830320 15863 raft_consensus.cc:686] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:02.896096 15844 raft_consensus.cc:686] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:02.845252 15865 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:02.932750 15187 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.028s sys 0.105s
I20260806 20:35:03.001282 15844 consensus_queue.cc:237] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:02.988660 15863 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:03.043567 15859 consensus_meta.cc:287] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.524s user 0.006s sys 0.008s
W20260806 20:35:03.048439 15850 consensus_meta.cc:287] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.884s user 0.017s sys 0.001s
W20260806 20:35:03.006549 15818 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.012s sys 0.032s
W20260806 20:35:03.113132 14368 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.992s user 0.000s sys 0.023s
W20260806 20:35:03.114416 14368 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.994s user 0.000s sys 0.024s
W20260806 20:35:03.132767 14518 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.007s sys 0.023s
W20260806 20:35:03.133432 14518 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.706s user 0.007s sys 0.023s
W20260806 20:35:03.134053 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 723770us
W20260806 20:35:03.157894 14631 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.011s sys 0.024s
W20260806 20:35:03.158607 14631 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.900s user 0.011s sys 0.025s
I20260806 20:35:03.070557 15850 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:03.070788 15818 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.012s sys 0.033s
I20260806 20:35:03.067348 15859 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:02.982291 15187 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.028s sys 0.106s
W20260806 20:35:03.176846 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.003s sys 0.009s
I20260806 20:35:03.180786 15710 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:03.224231 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.002s sys 0.004s
W20260806 20:35:03.523355 15183 consensus_meta.cc:287] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.700s user 0.012s sys 0.008s
I20260806 20:35:03.543625 15183 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:03.543969 15819 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:03.528098 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 90491.1us
I20260806 20:35:03.436342 15665 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.545073 15819 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.546545 15912 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:03.546970 15665 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.547418 15912 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.548856 15819 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.549018 15898 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.549537 15819 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.549925 15888 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.549752 15898 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.550529 15819 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.550954 15888 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.552556 15819 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.552867 15910 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.434597 15800 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.552943 15856 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.553784 15910 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.553901 15800 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.554114 15856 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.556301 15800 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:03.222338 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.666s user 0.003s sys 0.009s
I20260806 20:35:03.552197 15898 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:03.560282 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.985s user 0.009s sys 0.015s
I20260806 20:35:03.436049 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1361) took 1825 ms. Trace:
I20260806 20:35:03.561390 15910 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:03.562067 15754 consensus_meta.cc:287] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.913s user 0.012s sys 0.002s
W20260806 20:35:03.562505 14548 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.681s user 0.005s sys 0.005s
I20260806 20:35:03.563503 15856 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.553056 15819 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.593945 15754 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.597054 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1363) took 1978 ms. Trace:
I20260806 20:35:03.597628 15754 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.597565 14478 rpcz_store.cc:276] 0806 20:35:01.618673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.708328 (+ 89655us) service_pool.cc:227] Handling call
0806 20:35:01.708709 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:01.919512 (+210803us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:01.919542 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:01.919578 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:01.919606 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.596494 (+1676888us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54003336d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:03.596574 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:03.596878 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":484250368}
I20260806 20:35:03.598649 15819 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.599647 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:03.601109 14469 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:03.601325 15819 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:03.601650 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1359) took 1991 ms (client timeout 1834 ms). Trace:
I20260806 20:35:03.525765 15710 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.615064 15710 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.621838 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1077) took 2518 ms. Trace:
I20260806 20:35:03.622344 14546 rpcz_store.cc:276] 0806 20:35:01.103665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.103806 (+ 141us) service_pool.cc:227] Handling call
0806 20:35:01.104125 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:01.350346 (+246221us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:01.374609 (+ 24263us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:01.406013 (+ 31404us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:01.409214 (+ 3201us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.621149 (+2211935us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540018ecd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:03.621229 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:03.621540 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":503532800}
I20260806 20:35:03.624298 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 315) took 1398 ms. Trace:
I20260806 20:35:03.620018 15739 raft_consensus.cc:3032] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:03.625238 15739 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:03.624774 15185 rpcz_store.cc:276] 0806 20:35:02.225424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:02.229924 (+ 4500us) service_pool.cc:227] Handling call
0806 20:35:02.278821 (+ 48897us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:02.465131 (+186310us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.623540 (+1158409us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54003d0ed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:03.623985 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":364295936}
I20260806 20:35:03.564872 15859 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.524137 15908 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.435870 15818 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.434317 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.407212 15796 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:03.227128 15909 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260806 20:35:03.407716 15765 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.578989 15800 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:03.602190 14469 rpcz_store.cc:269] 0806 20:35:01.609830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.616444 (+ 6614us) service_pool.cc:227] Handling call
0806 20:35:03.599090 (+1982646us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:03.601545 (+ 2455us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5377408}
I20260806 20:35:03.223944 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:03.651211 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 1.07415e+06us
I20260806 20:35:03.658538 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 69) took 1418 ms. Trace:
I20260806 20:35:03.656384 15818 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.659018 14631 rpcz_store.cc:276] 0806 20:35:02.240531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:02.244809 (+ 4278us) service_pool.cc:227] Handling call
0806 20:35:02.257737 (+ 12928us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:02.257944 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:03.184350 (+926406us) log.cc:842] Serialized 14 byte log entry
0806 20:35:03.657711 (+473361us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:03.657816 (+ 105us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:03.658038 (+ 222us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.658190 (+ 152us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:03.658448 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":667209,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":966,"raft.queue_time_us":131,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":260,"thread_start_us":1370872,"threads_started":2}
I20260806 20:35:03.577292 15778 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:03.654706 15885 raft_consensus.cc:979] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: : Instructing follower 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:35:03.673934 15710 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.685113 15800 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:03.688023 14442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.299s user 0.006s sys 0.034s
I20260806 20:35:03.696445 15710 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.696873 15739 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.698585 15739 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:03.699926 15710 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.702533 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:03.700819 15825 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:03.721871 15825 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.725847 15796 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.726540 15825 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:03.730660 14442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.454s user 0.007s sys 0.035s
W20260806 20:35:03.731172 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 1.4543e+06us
I20260806 20:35:03.715194 15888 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.735733 15778 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.636787 14474 rpcz_store.cc:276] 0806 20:35:01.610751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.611562 (+ 811us) service_pool.cc:227] Handling call
0806 20:35:01.611974 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:01.612343 (+ 369us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:02.718918 (+1106575us) log.cc:842] Serialized 14 byte log entry
0806 20:35:03.435063 (+716145us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:03.435169 (+ 106us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:03.435368 (+ 199us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.435544 (+ 176us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:03.435749 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56947,"raft.queue_time_us":58382,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":41820,"thread_start_us":443849,"threads_started":1,"wal-append.queue_time_us":449164}
I20260806 20:35:03.738525 15778 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:03.739986 15833 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.740761 15833 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.741163 15740 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.741415 15765 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.741963 15740 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.743449 15833 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.744088 15859 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:03.744215 15912 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.744580 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 101) took 1088 ms. Trace:
I20260806 20:35:03.741036 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:03.744810 15859 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.689553 15820 raft_consensus.cc:1067] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Signalling peer 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:35:03.745405 15796 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.744377 15740 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.745508 15186 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:03.689523 15908 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.747087 15859 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.745070 14548 rpcz_store.cc:276] 0806 20:35:02.655878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:02.684708 (+ 28830us) service_pool.cc:227] Handling call
0806 20:35:02.695501 (+ 10793us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:02.695905 (+ 404us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:03.596681 (+900776us) log.cc:842] Serialized 14 byte log entry
0806 20:35:03.652790 (+ 56109us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:03.661065 (+ 8275us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:03.678358 (+ 17293us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.687129 (+ 8771us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:03.744504 (+ 57375us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46513,"prepare.queue_time_us":870279,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":93,"raft.queue_time_us":53260,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":6401,"thread_start_us":894584,"threads_started":1,"wal-append.queue_time_us":20792}
I20260806 20:35:03.749445 15816 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:03.750257 15816 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.752570 15800 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.752693 15908 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.714741 15665 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.755780 14518 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:03.763471 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:03.764699 15179 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:03.765991 15850 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:03.773437 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 227102us
I20260806 20:35:03.766957 15852 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:03.761762 15912 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:03.770556 15816 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.775281 15850 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.778602 15825 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:03.781235 15825 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:03.781561 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 79367.2us
I20260806 20:35:03.781364 15850 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.783135 14595 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:03.783820 15850 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.776068 15820 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:03.781773 15825 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:03.786860 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:03.789085 15181 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:03.790571 14441 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:03.787029 15765 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:03.785634 14475 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:03.793771 15920 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:03.796265 15920 raft_consensus.cc:686] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:03.784982 15912 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.796504 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.798363 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.798503 15852 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.804979 14552 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:03.805042 15908 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:03.815316 14547 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:03.818894 14555 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:03.808893 15875 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:03.817731 15908 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.822216 14518 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:03.813948 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:03.824785 15908 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.825271 15837 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:03.825783 15837 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.826201 15837 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:03.812757 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:03.830931 15185 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:03.832406 14518 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:03.834249 14441 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:03.834512 15898 consensus_queue.cc:1046] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:03.835183 15888 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:03.835446 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 95) took 1540 ms. Trace:
I20260806 20:35:03.835629 15888 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:03.836024 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 53160.9us
I20260806 20:35:03.836306 15800 raft_consensus.cc:3032] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:03.836879 15800 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:03.837740 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:03.812934 15893 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.810s user 0.011s sys 0.008s
I20260806 20:35:03.838318 15800 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.835850 14553 rpcz_store.cc:276] 0806 20:35:02.295324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:02.295505 (+ 181us) service_pool.cc:227] Handling call
0806 20:35:02.368020 (+ 72515us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:02.386040 (+ 18020us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.835020 (+1448980us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b54002da0d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:03.835276 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39273216}
I20260806 20:35:03.841881 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60780 (request call id 273) took 1441 ms. Trace:
I20260806 20:35:03.843770 14441 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:03.836076 15888 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:03.848389 14474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:35:03.831820 15912 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.843421 15187 rpcz_store.cc:276] 0806 20:35:02.398678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:02.400294 (+ 1616us) service_pool.cc:227] Handling call
0806 20:35:02.400716 (+ 422us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:02.403794 (+ 3078us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:02.403832 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:02.404277 (+ 445us) log.cc:842] Serialized 26 byte log entry
0806 20:35:03.591165 (+1186888us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:03.627510 (+ 36345us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.630777 (+ 3267us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:03.807384 (+176607us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b2400016e38. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0423cf4 00007fb9b0421dfb 00007fb9b04b0559 00007fb9b04b01e5 00007fb9b04af82d 00007fb9b315e46e 00007fb9b3175c0c 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551
0806 20:35:03.839653 (+ 32269us) raft_consensus.cc:1671] finished
0806 20:35:03.839671 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:03.839875 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14197,"raft.queue_time_us":10720,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7862,"spinlock_wait_cycles":19483520,"thread_start_us":816459,"threads_started":1,"wal-append.queue_time_us":587963}
I20260806 20:35:03.851400 15922 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.851511 14553 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:03.852942 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.850780 15815 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.854336 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.854031 15815 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.849411 14474 raft_consensus.cc:595] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:35:03.855131 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.852218 15922 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.858008 15849 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:03.851579 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.887930 15912 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:03.855386 15908 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 20:35:03.891176 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 64113.4us
I20260806 20:35:03.888821 15912 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.861138 15800 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:03.913807 14549 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:03.918007 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.918442 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.918584 15778 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:03.805073 15920 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.940774 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.893060 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.944133 14551 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:03.951995 15811 raft_consensus.cc:3032] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:03.952414 15811 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:03.860301 15849 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:03.892621 15908 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.964010 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:03.964216 15918 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:03.953578 15811 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:03.965623 14546 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:03.971935 15815 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:03.973696 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 51724.8us
I20260806 20:35:03.889418 15922 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.941469 15852 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:03.980311 15811 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:03.982359 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 144805us
I20260806 20:35:03.980804 15842 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:03.972828 15908 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:03.985971 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:03.991501 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:03.993290 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:03.994846 15797 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:03.974645 15179 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:03.990935 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 140804us
I20260806 20:35:04.001048 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 318) took 1769 ms. Trace:
I20260806 20:35:04.008114 15849 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.008685 15692 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:03.995659 15797 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.009374 15692 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.011161 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.011890 14552 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:04.016317 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:04.012660 15918 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.012365 15692 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.020133 15852 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:04.020396 14630 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:04.020623 15879 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:04.022264 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:04.028569 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.028853 14552 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.029403 14626 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:04.030295 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.020339 14595 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:04.030419 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.031379 14552 raft_consensus.cc:2379] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:04.017295 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.055424 15797 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.064271 15850 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:04.005515 15842 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.013372 15182 rpcz_store.cc:276] 0806 20:35:02.226496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:02.229822 (+ 3326us) service_pool.cc:227] Handling call
0806 20:35:03.595318 (+1365496us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593bce 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:03.595705 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:03.728722 (+133017us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:03.728773 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:03.728849 (+ 76us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:03.728919 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:03.995567 (+266648us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54003787d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:03.995637 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:03.995777 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":350889088}
W20260806 20:35:04.089393 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 62352.1us
I20260806 20:35:04.103091 15151 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:04.112879 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.045260 15864 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:04.171110 15798 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:04.172041 15798 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.175222 15798 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.114253 15879 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.164423 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.091145 14546 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:04.044862 15912 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:04.183844 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 163676us
W20260806 20:35:04.184446 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 169006us
I20260806 20:35:04.184798 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.186256 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.159019 15852 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:04.176317 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 157900us
I20260806 20:35:04.192360 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.195251 14478 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:04.192248 14554 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17eec21790894c559ad9623765626416"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:54508
I20260806 20:35:04.197775 14554 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:35:04.198181 14554 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.198042 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.195436 14441 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:04.182730 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.186372 15849 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:04.202256 15847 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:04.205219 14518 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:04.190675 15797 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:04.200526 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:04.207350 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 64464.3us
I20260806 20:35:04.210901 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.224814 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.188805 15864 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.232643 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.233381 14630 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:04.225391 14628 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:04.228503 15799 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:04.228561 15827 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.230588 15850 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:04.217532 15835 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.187589 15739 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260806 20:35:04.217571 15847 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.234757 14626 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:04.236871 14444 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:04.197849 15912 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:04.241521 15853 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.234330 15849 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.205309 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.231729 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.217991 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.227195 14595 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:04.242975 15827 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.236220 15799 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:04.247874 15850 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:04.246495 15835 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.248847 15918 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.271536 15799 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:04.252745 15842 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.276968 15153 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:04.277614 15847 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:04.275134 15835 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.278168 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.278766 15818 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.278970 14630 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:04.279336 15153 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:04.278210 15853 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.280270 15743 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:04.280850 15849 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.281634 14631 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:04.282037 15853 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.252406 15797 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.283995 15869 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:04.284695 14478 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:04.283363 15842 raft_consensus.cc:979] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: : Instructing follower 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:35:04.285286 15836 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:04.285619 15797 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.286134 15860 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:04.287276 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 86252.5us
I20260806 20:35:04.286506 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.292367 15743 raft_consensus.cc:1067] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Signalling peer 148bf2a11dba423ab364371f0458e871 to start an election
I20260806 20:35:04.270121 15827 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.274305 15879 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.299746 14629 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:04.293998 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:04.292307 15722 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:04.309273 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.311085 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.323745 15849 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.324555 15911 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.297784 15912 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.333643 15179 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:59384
I20260806 20:35:04.325124 15911 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:04.313450 15181 consensus_meta.cc:287] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.011s
I20260806 20:35:04.334328 15179 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:35:04.334525 15181 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:04.334775 15179 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.324389 15849 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.289073 15153 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:04.372951 14441 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:04.374053 14626 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:04.376223 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 64143.6us
I20260806 20:35:04.400664 15860 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.400923 14368 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45970 (request call id 231) took 2735 ms. Trace:
I20260806 20:35:04.401235 15860 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:04.401680 15860 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:04.372956 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 88647.4us
I20260806 20:35:04.429761 14595 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:04.430826 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 57974.4us
W20260806 20:35:04.439313 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 121363us
I20260806 20:35:04.325503 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:04.475090 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 252779us
W20260806 20:35:04.475945 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 284040us
I20260806 20:35:04.476892 15864 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.476738 15182 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:04.449954 15920 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:04.481634 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:04.489075 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.498535 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.499264 14630 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
W20260806 20:35:04.506487 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 76888.6us
I20260806 20:35:04.484033 15915 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:04.523924 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:04.488857 15920 raft_consensus.cc:686] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:04.538691 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 57243.4us
I20260806 20:35:04.539130 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.520357 15914 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.549753 15739 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 20:35:04.555819 15888 consensus_meta.cc:287] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.695s user 0.007s sys 0.008s
I20260806 20:35:04.555971 15821 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:04.556576 14444 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:04.569432 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.570197 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.557272 15911 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.538146 15920 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.576798 15187 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.575258 14629 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:04.562657 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:04.576432 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.572755 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.401418 14368 rpcz_store.cc:276] 0806 20:35:01.665493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:01.665642 (+ 149us) service_pool.cc:227] Handling call
0806 20:35:04.400814 (+2735172us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:02.992223 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:02.994357 (+ 2134us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:02.994410 (+ 53us) write_op.cc:435] Acquired schema lock
0806 20:35:02.994439 (+ 29us) tablet.cc:660] Decoding operations
0806 20:35:02.995304 (+ 865us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:02.995440 (+ 136us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:02.995511 (+ 71us) tablet.cc:683] Acquiring locks for 4 operations
0806 20:35:02.996061 (+ 550us) tablet.cc:699] Row locks acquired
0806 20:35:02.996481 (+ 420us) write_op.cc:260] PREPARE: finished
0806 20:35:02.996692 (+ 211us) write_op.cc:270] Start()
0806 20:35:02.996843 (+ 151us) write_op.cc:276] Timestamp: P: 1786048502996631 usec, L: 0
0806 20:35:02.996875 (+ 32us) op_driver.cc:331] REPLICATION: starting
0806 20:35:02.997777 (+ 902us) log.cc:842] Serialized 1792 byte log entry
0806 20:35:03.084914 (+ 87137us) op_driver.cc:444] REPLICATION: finished
0806 20:35:03.086793 (+ 1879us) write_op.cc:301] APPLY: starting
0806 20:35:03.086907 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:03.087110 (+ 203us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:03.087443 (+ 333us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:03.088108 (+ 665us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:03.088333 (+ 225us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0806 20:35:03.088380 (+ 47us) write_op.cc:312] APPLY: finished
0806 20:35:03.088673 (+ 293us) log.cc:842] Serialized 41 byte log entry
0806 20:35:03.088851 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:03.175062 (+ 86211us) write_op.cc:454] Released schema lock
0806 20:35:03.175208 (+ 146us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:03.175806 (+ 598us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:03.175847 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":1403,"apply.run_cpu_time_us":4186,"apply.run_wall_time_us":89611,"num_ops":4,"prepare.queue_time_us":867529,"prepare.run_cpu_time_us":7733,"prepare.run_wall_time_us":65667,"raft.queue_time_us":29006,"raft.run_cpu_time_us":1255,"raft.run_wall_time_us":3976,"replication_time_us":87923,"thread_start_us":1062803,"threads_started":4,"wal-append.queue_time_us":55093,"wal-append.run_cpu_time_us":6695,"wal-append.run_wall_time_us":1122847}]]}
I20260806 20:35:04.580651 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:04.644135 14475 consensus_meta.cc:287] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.825s user 0.023s sys 0.013s
W20260806 20:35:04.644767 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 56188.7us
I20260806 20:35:04.646960 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.563380 15888 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.580461 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.592971 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.590745 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.577971 15821 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:04.596380 15825 consensus_meta.cc:287] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.794s user 0.021s sys 0.066s
I20260806 20:35:04.662181 14630 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:04.580062 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.584949 15912 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.589682 15882 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:04.689147 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.590867 15185 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:04.691406 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 142529us
I20260806 20:35:04.688490 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.694051 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.658216 15849 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.648732 14475 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:04.590364 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.592602 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.704054 14444 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:04.695027 15800 consensus_meta.cc:287] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.739s user 0.015s sys 0.014s
I20260806 20:35:04.708432 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.713752 14470 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:04.709160 15723 raft_consensus.cc:1067] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Signalling peer 8e0cfd35f2994f91aaae46c25c6d6bff to start an election
I20260806 20:35:04.709766 15153 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:04.710249 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.717720 14477 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
W20260806 20:35:04.700800 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 44921us
I20260806 20:35:04.690496 15825 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.570847 15914 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.726864 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.728847 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.729463 14477 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:04.700052 15835 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:04.723917 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1345) took 4598 ms (client timeout 1905 ms). Trace:
I20260806 20:35:04.725930 15800 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:04.740886 14554 consensus_meta.cc:287] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.534s user 0.014s sys 0.003s
I20260806 20:35:04.741389 14626 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:04.744374 15794 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:04.723485 15723 raft_consensus.cc:1070] unable to start election on peer 8e0cfd35f2994f91aaae46c25c6d6bff: Not found: unknown peer
I20260806 20:35:04.745462 14554 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.736351 14547 raft_consensus.cc:1226] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.750125 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:04.751236 15837 consensus_meta.cc:287] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.923s user 0.022s sys 0.045s
I20260806 20:35:04.761858 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.760831 14477 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:04.761050 14554 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.768082 14552 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.768540 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.770651 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.771003 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.771799 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.773110 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.773576 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.774364 14442 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:04.776686 14442 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:04.776983 15743 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:04.777644 15692 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.778021 15692 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.778406 15692 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.778986 15743 consensus_queue.cc:1046] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:35:04.781361 15743 raft_consensus.cc:3032] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:04.781786 15743 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:04.783041 15743 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.783499 14442 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:04.784294 15879 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.784791 15879 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.785271 15879 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.785422 14442 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:04.785378 15821 consensus_queue.cc:1046] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:04.786679 15821 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.787024 15821 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.787367 15821 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.787559 15898 raft_consensus.cc:3032] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:04.788049 15898 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:04.788925 15898 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.790951 15808 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:04.791646 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.791514 15808 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.792868 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:04.797183 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 43687.7us
W20260806 20:35:04.764690 15908 consensus_meta.cc:287] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.747s user 0.028s sys 0.044s
I20260806 20:35:04.790501 15898 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:04.796375 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.800273 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.800742 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:04.801311 14476 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 20:35:04.814822 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:04.816120 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.765383 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.824112 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:04.822101 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 261534us
I20260806 20:35:04.787322 15692 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.801864 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.812747 15835 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.833033 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:04.765303 15837 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.799669 15888 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:04.742969 14475 rpcz_store.cc:269] 0806 20:35:00.120435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:00.123599 (+ 3164us) service_pool.cc:227] Handling call
0806 20:35:00.791634 (+668035us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54003dc820. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b3483730 00007fb9b34f6eff 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:01.570641 (+779007us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b180013a2e0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b352e856 00007fb9b3480f7e 00007fb9b34ea367 00007fb9b34ebb4e 00007fb9b34f7228 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c
0806 20:35:03.774407 (+2203766us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b58003800b0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b346ef63 00007fb9b34d2aba 00007fb9b34ea3af 00007fb9b34ebb4e 00007fb9b34f7228 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c
0806 20:35:04.718651 (+944244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3071139840}
I20260806 20:35:04.821288 15842 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.837802 14518 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:04.834515 15811 consensus_meta.cc:287] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.854s user 0.011s sys 0.007s
I20260806 20:35:04.813861 15908 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:04.835120 15928 log.cc:924] Time spent T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260806 20:35:04.832242 15743 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:04.844759 14553 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.851548 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.849031 15184 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:04.851691 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.822019 15179 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.854328 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.838383 15183 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:04.856689 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:04.856689 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 143561us
I20260806 20:35:04.841955 15842 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:04.841919 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:04.842080 15825 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.863488 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.848548 15912 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.864749 14474 raft_consensus.cc:2379] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 2: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:04.865638 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:04.866309 14473 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.866719 15692 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.867296 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:04.869271 15186 raft_consensus.cc:1226] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.870364 14548 raft_consensus.cc:595] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:35:04.871657 14470 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.843015 15888 consensus_queue.cc:1046] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260806 20:35:04.873931 14626 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.854823 15914 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.863113 15842 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.877931 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.878165 14521 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:04.871420 15185 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.883935 14521 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:04.888125 15853 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.888526 15853 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:04.888924 15853 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.892968 14470 raft_consensus.cc:1226] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:04.894620 15150 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:04.884153 15827 consensus_meta.cc:287] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.581s user 0.011s sys 0.000s
I20260806 20:35:04.890151 14521 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:04.910348 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 91964us
W20260806 20:35:04.912185 14595 connection.cc:232] RPC call timeout handler was delayed by -1.2566547890310176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1889s for a response
I20260806 20:35:04.913149 14595 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:04.911158 15827 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:04.917264 15911 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:04.917675 15911 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:04.918049 15911 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:04.918700 15827 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.921214 15837 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:04.922998 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.923468 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.925864 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.927984 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.919169 15836 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260806 20:35:04.930191 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.915369 15835 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:04.933580 14477 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:04.886008 15933 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:04.939594 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.941062 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.933605 15835 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:04.942488 15835 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:04.942742 15939 consensus_queue.cc:1046] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:04.943888 15939 raft_consensus.cc:3032] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:04.944175 15939 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:04.944835 15908 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.945149 15939 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:04.946336 15939 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:04.946437 15179 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:04.897143 15888 consensus_queue.cc:1225] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer 8e0cfd35f2994f91aaae46c25c6d6bff log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:35:04.948437 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:04.964854 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:04.972968 14469 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:04.976009 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54494 (request call id 809) took 4224 ms. Trace:
I20260806 20:35:04.936425 15933 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:04.973062 15882 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:04.976430 14479 rpcz_store.cc:276] 0806 20:35:00.751400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:00.751542 (+ 142us) service_pool.cc:227] Handling call
0806 20:35:04.975253 (+4223711us) spinlock_profiling.cc:243] Waited 4.04 s on lock 0x7b54001176d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b8438c6e 00007fb9b1b2f807 00007fb9b1b2f792 00007fb9b1b2f6c1 00007fb9b1b2f63c 00007fb9b1b2e8f2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:04.975774 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9292810112}
I20260806 20:35:04.947500 15150 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:04.979308 15836 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:04.976982 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:04.979900 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:04.980574 15178 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:04.977705 15882 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:04.961030 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 145248us
I20260806 20:35:04.987221 14442 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:04.990173 14521 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:04.961867 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 48852.3us
I20260806 20:35:05.003643 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.010255 14595 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.013423 15808 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:04.990916 15723 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.001612 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:05.015663 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:04.988526 15933 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.021389 15184 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:05.020432 14521 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:05.025173 14551 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:05.026801 15860 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.625s user 0.013s sys 0.006s
W20260806 20:35:05.007017 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 98601.3us
W20260806 20:35:05.028558 15850 consensus_meta.cc:287] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.741s user 0.005s sys 0.016s
I20260806 20:35:05.028925 15185 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:05.028962 14548 raft_consensus.cc:595] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:35:04.999779 15849 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:05.017844 15936 consensus_queue.cc:1046] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260806 20:35:05.007767 15879 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.032866 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.033893 15877 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.036816 15827 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:05.037132 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.040267 15934 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.056535 14595 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.047209 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.055193 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:05.045480 14548 raft_consensus.cc:603] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:05.050204 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.050666 15936 raft_consensus.cc:3032] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
W20260806 20:35:05.051492 15799 consensus_meta.cc:287] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.758s user 0.008s sys 0.048s
I20260806 20:35:05.063107 15799 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.050130 15181 raft_consensus.cc:1226] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.063818 15879 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.061954 15936 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:05.065150 15799 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:05.053207 14441 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:05.065414 15936 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.066749 15936 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.043062 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:05.050130 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.057497 14477 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:05.059055 14470 raft_consensus.cc:1226] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.070333 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.058480 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1109) took 1240 ms. Trace:
I20260806 20:35:05.060941 14475 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:05.060889 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.045104 15849 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:05.038031 15852 consensus_queue.cc:1046] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260806 20:35:05.076444 14471 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:05.090873 15179 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:35:05.086603 15849 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:05.076627 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.077105 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.077368 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.077191 15877 consensus_queue.cc:1046] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:05.078083 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.078212 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.078963 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.079195 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1355) took 4332 ms. Trace:
I20260806 20:35:05.080346 15722 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.084123 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.094668 14553 raft_consensus.cc:2379] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:05.053922 15827 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:05.093127 14476 rpcz_store.cc:276] 0806 20:35:00.746450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:00.746577 (+ 127us) service_pool.cc:227] Handling call
0806 20:35:04.705458 (+3958881us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:04.813511 (+108053us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:05.078127 (+264616us) spinlock_profiling.cc:243] Waited 93.6 ms on lock 0x7b54001176d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:05.078974 (+ 847us) spinlock_profiling.cc:243] Waited 3.94 s on lock 0x7b54001176d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:05.079120 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9274304640}
I20260806 20:35:05.045291 15850 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.038916 15860 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.099120 15827 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.099824 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.100684 15186 raft_consensus.cc:2379] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:05.101392 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.075747 15799 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:05.074030 14547 rpcz_store.cc:276] 0806 20:35:03.818250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:03.820906 (+ 2656us) service_pool.cc:227] Handling call
0806 20:35:03.821392 (+ 486us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:04.762735 (+941343us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:05.055152 (+292417us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54000861d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:05.058413 (+ 3261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2081575808}
I20260806 20:35:05.102445 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.117168 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:05.117470 14477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:35:05.124466 14477 raft_consensus.cc:595] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:35:05.125070 14477 raft_consensus.cc:603] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:35:05.116328 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 64031.8us
I20260806 20:35:05.119998 15150 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.119455 15827 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.126719 15921 consensus_queue.cc:1046] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:05.127225 15827 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.127926 15921 raft_consensus.cc:3032] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.128340 15921 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:05.121439 14476 raft_consensus.cc:2362] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2. Re-sending same reply.
I20260806 20:35:05.129367 15921 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.130733 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.131049 14550 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:05.131727 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.093009 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.130702 15921 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.135187 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:05.136116 15182 raft_consensus.cc:2379] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:05.136545 15799 raft_consensus.cc:686] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.138053 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:05.137775 15799 consensus_queue.cc:237] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.140769 15868 raft_consensus.cc:3032] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.141196 15868 raft_consensus.cc:729] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:05.142132 15868 consensus_queue.cc:260] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.143275 14631 raft_consensus.cc:1261] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:05.144176 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.133198 15853 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.145889 14595 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:05.147001 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 126697us
I20260806 20:35:05.144850 15847 raft_consensus.cc:3032] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.149206 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.149837 15847 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:05.149887 15852 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.115003 15821 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:05.157433 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 98760.6us
I20260806 20:35:05.158018 14595 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.111722 15860 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.167186 15184 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:05.135231 15850 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:05.170830 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:05.149212 15692 raft_consensus.cc:3032] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.174134 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.170994 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.175334 14478 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.173846 15692 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.177127 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.177220 15912 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.176970 15692 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.178006 15692 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.161824 15836 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.178891 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:05.184532 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 64675.2us
W20260806 20:35:05.187785 14421 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet a94bd288a57145e8b254611b1dd82586
I20260806 20:35:05.187893 15150 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:05.188264 14421 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet a94bd288a57145e8b254611b1dd82586
I20260806 20:35:05.188690 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 61) took 1157 ms. Trace:
I20260806 20:35:05.189330 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.189608 14473 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.157013 15882 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.189076 14474 rpcz_store.cc:276] 0806 20:35:04.041142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:04.055694 (+ 14552us) service_pool.cc:227] Handling call
0806 20:35:05.188494 (+1132800us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b54002319d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:05.188638 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1509450624}
I20260806 20:35:05.168578 15852 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:05.191076 15852 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:05.191380 15882 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:05.193284 15934 consensus_queue.cc:1046] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:05.194974 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.188889 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.192157 15849 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.196727 14518 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.196556 14521 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:05.197491 15800 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.205122 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.206336 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.207541 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.210196 15821 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.210420 15835 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.191473 15852 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:05.211690 14471 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.192297 14473 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.214612 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.224541 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.225353 14626 raft_consensus.cc:3032] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.225719 14626 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:05.226480 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.224630 15864 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.194372 15934 raft_consensus.cc:3032] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.227627 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.229382 14470 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:05.230253 15912 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:05.230778 15912 raft_consensus.cc:686] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:05.230875 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.231721 15912 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.235639 14441 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.239360 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.219762 15184 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.241870 14441 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.211684 15849 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.247531 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.232897 14626 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.254900 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.255980 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:05.237025 15943 log.cc:924] Time spent T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260806 20:35:05.248879 15808 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.233464 15934 raft_consensus.cc:729] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:05.267290 15934 consensus_queue.cc:260] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.268484 15934 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.230792 15180 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.258648 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.260932 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.271783 15184 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:05.280788 14625 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.280813 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.233268 15882 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:05.228562 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:05.259316 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 89612.9us
I20260806 20:35:05.264910 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.265448 14470 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.266078 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:05.286150 14479 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:05.266304 15835 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.269361 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.199375 15842 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.245395 15868 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.272624 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.215633 14369 catalog_manager.cc:5495] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: leader changed from <none> to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 2 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:05.211614 15847 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.286645 14518 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.259732 15888 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.287336 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.292650 15821 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.292510 15743 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:05.295007 14444 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.295436 15182 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:05.262746 15179 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:05.301108 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.255086 15836 consensus_queue.cc:1046] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20260806 20:35:05.303880 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.272779 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.308238 15945 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:05.308826 15945 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.309405 14549 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.311007 15945 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.311666 14444 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.312150 15945 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.312392 15808 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.313208 15853 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.314682 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.315938 14553 raft_consensus.cc:1226] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.316689 14469 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:05.317647 14626 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.319792 14629 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:05.321982 15911 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.324542 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.326185 14478 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:05.329979 14470 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
W20260806 20:35:05.338354 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 176199us
I20260806 20:35:05.340533 15882 raft_consensus.cc:686] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:05.261408 15933 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.330770 14369 catalog_manager.cc:5495] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: config changed from index 2 to 3, 148bf2a11dba423ab364371f0458e871 (127.12.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:05.304078 15723 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.349059 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.349550 14552 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.349714 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.353631 15151 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.383517 15151 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.349603 14550 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:05.328924 15836 raft_consensus.cc:3032] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:05.384788 14477 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:05.385591 14476 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:05.389433 14555 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.342733 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:05.395615 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 80994.7us
I20260806 20:35:05.395850 14554 raft_consensus.cc:1226] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.398667 14551 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:35:05.400687 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.398689 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:05.401136 15692 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.383101 14474 raft_consensus.cc:1261] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:05.408241 14625 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:05.412230 15153 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:05.388064 15811 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.389034 15836 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:05.412245 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 125703us
I20260806 20:35:05.413386 14444 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.434948 14627 raft_consensus.cc:1226] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.435917 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.437389 14444 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.439241 14518 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.440491 14518 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.404688 15860 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.442271 14518 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.401404 14547 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.393985 15821 consensus_queue.cc:1046] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 20:35:05.461557 14518 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.399945 15723 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260806 20:35:05.450060 15882 consensus_queue.cc:237] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.444037 15811 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:05.449203 15853 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.411826 15908 consensus_queue.cc:1046] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260806 20:35:05.429143 15864 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260806 20:35:05.461041 15799 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.470993 15911 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:05.449028 15691 consensus_queue.cc:1046] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.153s
I20260806 20:35:05.482525 15930 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.462549 15850 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.485874 15908 raft_consensus.cc:3032] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.486364 15908 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.487601 15953 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:05.487350 15908 consensus_queue.cc:260] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.488157 15953 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.449540 15836 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.490790 15821 raft_consensus.cc:3032] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.491190 15821 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.492231 15821 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.460953 15800 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.502341 14471 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.449667 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.480623 14444 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.444355 15692 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.441445 15898 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 20:35:05.506685 15811 raft_consensus.cc:686] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:05.519176 15150 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:05.520670 15150 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.521464 14548 raft_consensus.cc:1226] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:05.523097 15794 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:05.523593 15794 raft_consensus.cc:686] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:05.524140 15888 raft_consensus.cc:3032] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.526605 15888 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.453455 15952 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.527520 15888 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.529146 15860 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:05.529498 15888 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.499087 15933 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:05.525630 15947 consensus_queue.cc:1159] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260806 20:35:05.524484 15794 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.528179 15800 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.497107 15722 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:05.535084 15722 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.444177 15939 consensus_queue.cc:1046] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20260806 20:35:05.568315 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 45239.5us
W20260806 20:35:05.568734 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 88194.4us
I20260806 20:35:05.569504 14444 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:05.569725 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 108245us
I20260806 20:35:05.571941 14518 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.572024 14626 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:05.579893 14518 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:05.578615 15842 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:05.586596 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 203197us
W20260806 20:35:05.585511 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 179849us
I20260806 20:35:05.593408 15944 consensus_queue.cc:1046] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:05.597261 15948 raft_consensus.cc:3032] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.597846 15948 raft_consensus.cc:729] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:05.601657 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.602294 14629 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:05.605967 14444 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.569123 15836 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:05.568346 15908 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.574014 15847 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.613094 15835 consensus_queue.cc:1046] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:05.606976 14521 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:05.588376 15957 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.624485 14444 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:05.629071 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 216915us
I20260806 20:35:05.609586 15821 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.634887 15930 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:05.635807 15853 consensus_queue.cc:1046] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:05.599383 15948 consensus_queue.cc:260] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.651885 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.652874 14478 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.653237 15948 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.583000 15938 consensus_queue.cc:1046] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 20:35:05.632179 15743 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.641142 15953 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:05.645529 14444 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.646497 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.607686 15722 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:05.620950 15842 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.650352 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.681064 14444 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.630272 15811 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.651693 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:05.656580 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.658613 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.606669 15800 consensus_queue.cc:1046] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260806 20:35:05.633097 15912 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:05.693655 15912 raft_consensus.cc:686] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:05.659801 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.659968 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:05.664369 15953 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.705747 14547 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:05.665419 14595 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.669703 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.672420 15930 raft_consensus.cc:686] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:05.676931 15723 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.710734 15852 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.678463 15743 raft_consensus.cc:3032] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.712930 15743 raft_consensus.cc:729] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.678767 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.682222 14444 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.684265 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.648053 15939 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:05.687588 15920 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.704946 14554 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:05.708185 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.689288 15799 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 20:35:05.721377 15957 raft_consensus.cc:3032] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:05.729444 14478 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:05.730728 15936 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:05.737622 15936 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:05.734966 15827 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260806 20:35:05.738322 15936 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:05.742125 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 59965us
I20260806 20:35:05.743026 14444 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.720780 15965 raft_consensus.cc:3032] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 20:35:05.745643 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 80294.9us
I20260806 20:35:05.746593 14595 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.729802 15957 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:05.747316 15692 raft_consensus.cc:3032] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:05.792435 15692 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.792737 15808 raft_consensus.cc:3032] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:05.793325 15808 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:05.793814 15692 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.794543 15808 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.795449 15692 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.798316 14547 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.749367 15965 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:05.802253 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 59263.2us
W20260806 20:35:05.803301 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 56326.8us
I20260806 20:35:05.720770 15930 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:05.805557 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 59098.1us
I20260806 20:35:05.806247 14595 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.744485 15939 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.815018 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.730890 15835 consensus_queue.cc:1046] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20260806 20:35:05.818452 14630 raft_consensus.cc:1261] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:05.824209 14444 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:05.824673 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 92749us
I20260806 20:35:05.827361 14475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5277702c39f453cb1063cf506183484"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:35:05.827991 14475 raft_consensus.cc:595] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.828342 14475 raft_consensus.cc:603] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:05.828997 14632 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:05.806874 15938 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.828250 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:05.837154 14595 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.838176 14595 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.840896 14629 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:05.848415 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 236827us
I20260806 20:35:05.850088 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:05.853400 14624 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:05.841014 15952 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260806 20:35:05.837731 15939 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.858265 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:05.823143 15953 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:05.860690 15953 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:05.861074 15953 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.861555 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:05.823251 15921 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:05.863804 14630 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:35:05.863714 14521 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:05.865450 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:05.728868 15912 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.873710 14518 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.873898 14595 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.875988 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.822320 15920 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260806 20:35:05.876904 15918 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.866991 15921 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:05.878367 14444 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.878474 14518 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.880930 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 20:35:05.862514 15934 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.890851 15920 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.892634 15961 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.881328 15955 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.877547 15827 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:05.863569 15808 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:05.889793 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.897190 14633 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:05.897840 14518 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.896363 15723 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260806 20:35:05.908674 15973 log.cc:924] Time spent T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.062s user 0.004s sys 0.000s
I20260806 20:35:05.910915 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:35:05.911301 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:05.912384 14625 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:05.897037 15921 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.897272 15852 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:05.903373 15849 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.905095 15847 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:05.911852 15962 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:05.915623 15827 raft_consensus.cc:686] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:05.874980 15936 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.924573 15915 consensus_queue.cc:1046] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:05.918996 15723 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260806 20:35:05.921500 15877 consensus_queue.cc:1046] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260806 20:35:05.919520 15918 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:05.922683 15961 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:05.954610 15918 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:05.962888 15918 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.941972 15827 consensus_queue.cc:237] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:05.964993 14470 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:05.872160 15957 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:05.957998 15961 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:05.975898 15961 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:05.984994 14518 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.987636 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.988721 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:05.993976 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:05.943852 15974 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260806 20:35:05.999898 14595 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:05.992332 14518 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:06.014760 14518 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:05.969316 15915 consensus_queue.cc:1046] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 20:35:06.009147 15185 raft_consensus.cc:1226] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.009862 15179 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:06.003412 15912 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:06.020558 15825 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:05.984345 15944 consensus_queue.cc:1046] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260806 20:35:06.057717 14595 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:06.057946 14518 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:06.154981 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 97129.4us
W20260806 20:35:06.147306 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 239704us
I20260806 20:35:06.161022 15933 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:06.165637 15933 raft_consensus.cc:686] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:06.165934 15977 log.cc:924] Time spent T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.317s user 0.001s sys 0.003s
I20260806 20:35:06.166930 15933 consensus_queue.cc:237] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.098237 15180 raft_consensus.cc:2379] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:06.156266 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 113767us
W20260806 20:35:06.150135 15982 log.cc:924] Time spent T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
I20260806 20:35:06.058650 15850 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:06.175876 14518 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:06.084623 15877 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:06.178102 14627 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:06.093974 15969 log.cc:924] Time spent T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.262s user 0.002s sys 0.000s
I20260806 20:35:06.179468 14625 raft_consensus.cc:1261] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:06.157908 15934 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:06.211632 15821 consensus_meta.cc:287] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.535s user 0.008s sys 0.003s
I20260806 20:35:06.129576 15743 consensus_queue.cc:260] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.212308 15850 raft_consensus.cc:686] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:06.245540 15964 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:06.246326 15964 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.251499 15842 raft_consensus.cc:3032] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:06.251945 15842 raft_consensus.cc:729] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:06.253037 15983 raft_consensus.cc:3032] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:06.252799 15842 consensus_queue.cc:260] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.253580 15983 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:06.254024 15842 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:06.236162 15849 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.265064 15850 consensus_queue.cc:237] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.267746 15921 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:06.271081 15978 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
W20260806 20:35:06.271926 15981 log.cc:924] Time spent T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20260806 20:35:06.259424 15936 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:06.287138 15921 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:06.272305 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1424) took 1037 ms. Trace:
I20260806 20:35:06.469609 15996 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:06.094079 14473 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:06.513826 14633 rpcz_store.cc:276] 0806 20:35:05.235265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.235440 (+ 175us) service_pool.cc:227] Handling call
0806 20:35:06.272059 (+1036619us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b540009f954. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:06.272199 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1456401792}
I20260806 20:35:06.470793 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:06.320901 15849 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:06.344444 15825 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:06.302531 15918 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.327441 15953 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.325685 15961 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.334913 15962 raft_consensus.cc:3032] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:06.309691 15964 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:06.769761 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:06.456125 15965 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.502104 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:06.316598 15983 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.511875 14628 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:06.774304 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:06.774643 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:06.514176 15996 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:06.554744 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 379451us
W20260806 20:35:06.569466 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 393259us
I20260806 20:35:06.777465 15983 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.777577 14595 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:06.778149 15882 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 20:35:06.599295 14441 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.017s sys 0.018s
W20260806 20:35:06.780629 14441 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.931s user 0.018s sys 0.018s
I20260806 20:35:06.615576 14476 raft_consensus.cc:2396] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:06.785074 15180 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:06.785610 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:06.786254 14475 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.634406 15993 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:06.787173 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:06.787055 15993 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.787730 14479 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.788801 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:06.636148 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:06.647437 15811 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:06.789232 15993 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:06.789633 15811 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.789575 14625 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:06.791105 15184 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.792378 15811 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:06.793555 14477 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.794895 14474 raft_consensus.cc:1226] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.665212 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1410) took 1007 ms. Trace:
I20260806 20:35:06.796638 14470 rpcz_store.cc:276] 0806 20:35:05.658227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.658409 (+ 182us) service_pool.cc:227] Handling call
0806 20:35:06.665079 (+1006670us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54003c5854. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:06.665162 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":608131456}
W20260806 20:35:06.663632 14631 consensus_meta.cc:287] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.568s user 0.011s sys 0.001s
I20260806 20:35:06.685750 15898 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:06.797562 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:06.797746 14631 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:06.798050 15898 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.799190 14633 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:06.799275 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1423) took 1565 ms. Trace:
I20260806 20:35:06.799707 14631 rpcz_store.cc:276] 0806 20:35:05.234379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.248790 (+ 14411us) service_pool.cc:227] Handling call
0806 20:35:06.799079 (+1550289us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400160654. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:06.799184 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938003200}
I20260806 20:35:06.692638 15722 raft_consensus.cc:3032] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:06.695436 14471 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:06.707715 14369 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45954 (request call id 233) took 2323 ms. Trace:
I20260806 20:35:06.800617 15722 raft_consensus.cc:729] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:06.706921 14368 catalog_manager.cc:5495] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:06.706921 14555 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.801445 15936 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:06.722179 16001 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:06.802486 14368 catalog_manager.cc:5495] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:06.802764 16001 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.803577 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:06.806267 16001 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:06.800956 14369 rpcz_store.cc:276] 0806 20:35:04.385063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:04.399467 (+ 14404us) service_pool.cc:227] Handling call
0806 20:35:06.707655 (+2308188us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:05.600664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:05.829461 (+228797us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:05.829572 (+ 111us) write_op.cc:435] Acquired schema lock
0806 20:35:05.829593 (+ 21us) tablet.cc:660] Decoding operations
0806 20:35:05.830296 (+ 703us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:05.830406 (+ 110us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:05.830450 (+ 44us) tablet.cc:683] Acquiring locks for 2 operations
0806 20:35:05.830822 (+ 372us) tablet.cc:699] Row locks acquired
0806 20:35:05.830854 (+ 32us) write_op.cc:260] PREPARE: finished
0806 20:35:05.831029 (+ 175us) write_op.cc:270] Start()
0806 20:35:05.831183 (+ 154us) write_op.cc:276] Timestamp: P: 1786048505830956 usec, L: 0
0806 20:35:05.831226 (+ 43us) op_driver.cc:331] REPLICATION: starting
0806 20:35:05.927710 (+ 96484us) log.cc:842] Serialized 1053 byte log entry
0806 20:35:06.315092 (+387382us) op_driver.cc:444] REPLICATION: finished
0806 20:35:06.334856 (+ 19764us) write_op.cc:301] APPLY: starting
0806 20:35:06.334985 (+ 129us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:06.336857 (+ 1872us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:06.337648 (+ 791us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:06.338577 (+ 929us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:06.338826 (+ 249us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 20:35:06.338901 (+ 75us) write_op.cc:312] APPLY: finished
0806 20:35:06.339190 (+ 289us) log.cc:842] Serialized 29 byte log entry
0806 20:35:06.339357 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:06.339454 (+ 97us) write_op.cc:454] Released schema lock
0806 20:35:06.339490 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:06.339755 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:06.339779 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":19157,"apply.run_cpu_time_us":3005,"apply.run_wall_time_us":5478,"num_ops":2,"prepare.queue_time_us":147856,"prepare.run_cpu_time_us":37740,"prepare.run_wall_time_us":386337,"raft.queue_time_us":11960,"raft.run_cpu_time_us":2885,"raft.run_wall_time_us":3658,"replication_time_us":483723,"thread_start_us":171302,"threads_started":4,"wal-append.queue_time_us":92048}]]}
I20260806 20:35:06.807049 16001 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:06.808842 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:06.809536 15898 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:06.810702 14548 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:06.810945 15825 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:06.811617 14549 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:06.811496 15825 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.812888 15996 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:06.801479 15722 consensus_queue.cc:260] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:06.727239 15975 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.879s user 0.000s sys 0.006s
I20260806 20:35:06.728377 15847 raft_consensus.cc:3032] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:06.821530 14469 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.821777 15847 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:06.823197 15847 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.736267 14547 raft_consensus.cc:1226] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.737901 15918 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:06.826018 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 353) took 1215 ms. Trace:
I20260806 20:35:06.827863 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:06.744372 15868 raft_consensus.cc:3032] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:06.746392 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 290) took 1096 ms. Trace:
I20260806 20:35:06.807737 16001 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.747799 15743 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:06.711987 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.013s sys 0.038s
I20260806 20:35:06.765377 15957 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:06.767562 15961 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:06.767644 15962 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:06.768196 14624 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.770911 15953 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:06.772656 14552 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.773680 14551 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:06.591786 15989 log.cc:924] Time spent T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.375s user 0.003s sys 0.000s
I20260806 20:35:06.835423 15965 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:06.786558 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1412) took 1124 ms. Trace:
I20260806 20:35:06.826679 15182 rpcz_store.cc:276] 0806 20:35:05.610739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.614233 (+ 3494us) service_pool.cc:227] Handling call
0806 20:35:05.641169 (+ 26936us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:05.641935 (+ 766us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:05.642114 (+ 179us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:05.642921 (+ 807us) log.cc:842] Serialized 26 byte log entry
0806 20:35:05.847433 (+204512us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:05.847485 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:05.847570 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:06.726412 (+878842us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:06.825490 (+ 99078us) raft_consensus.cc:1671] finished
0806 20:35:06.825536 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:06.825909 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4749,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":8990,"spinlock_wait_cycles":34577792,"thread_start_us":203925,"threads_started":1,"wal-append.queue_time_us":204377}
I20260806 20:35:06.862504 14476 rpcz_store.cc:276] 0806 20:35:05.661676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.700716 (+ 39040us) service_pool.cc:227] Handling call
0806 20:35:05.828123 (+127407us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:06.786387 (+958264us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54002175d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:06.786503 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1806699136}
I20260806 20:35:06.834990 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:06.790581 15993 consensus_queue.cc:1046] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:06.692408 15976 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:06.847609 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:06.865330 14470 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.819831 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:06.813879 15722 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.828159 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:06.826469 14471 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:06.848476 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:06.856074 15908 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:06.869266 14474 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:06.857141 14521 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:06.869832 15908 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.828812 14629 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 2: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:06.841243 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:06.873833 14473 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:06.873924 15868 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:06.875088 15825 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:06.863315 15962 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.837543 15847 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.816974 15184 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:06.877601 15153 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:06.864832 14546 raft_consensus.cc:2379] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:06.837319 15964 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260806 20:35:06.853156 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:06.884262 15811 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:06.885327 14479 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:06.886193 15918 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:06.887357 15993 raft_consensus.cc:3032] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:06.877736 15179 rpcz_store.cc:276] 0806 20:35:05.649622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.651121 (+ 1499us) service_pool.cc:227] Handling call
0806 20:35:06.746016 (+1094895us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54004510d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:06.746218 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":392113024}
I20260806 20:35:06.900735 15692 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:06.934636 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:06.935047 15692 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.934640 15993 raft_consensus.cc:729] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:06.937073 15692 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:06.937299 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 53665.6us
I20260806 20:35:06.937637 15692 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:06.937173 15993 consensus_queue.cc:260] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.938266 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:06.938215 15692 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.878345 15898 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:06.939901 15898 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:06.940413 14469 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:06.905611 15150 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:06.941354 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 50024.9us
I20260806 20:35:06.828526 15821 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
I20260806 20:35:06.942564 15898 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:06.905198 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:06.943593 14474 raft_consensus.cc:2379] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:06.943993 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:06.944756 14551 raft_consensus.cc:2379] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:06.907269 15852 consensus_queue.cc:1046] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:06.909111 15962 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:06.910297 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:06.913213 14442 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:06.914079 15882 consensus_queue.cc:1046] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:06.919708 15860 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:06.920655 15153 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:06.829660 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:06.924472 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:06.924293 14470 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:06.940162 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:06.894927 15868 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:06.946121 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:06.943794 16000 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:06.909117 16001 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:06.997850 15860 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:07.030696 15860 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:06.935359 14629 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:07.000172 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.035457 14555 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:06.999418 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.015458 16000 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:07.037887 16000 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:07.038808 14469 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:07.003415 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 93198us
I20260806 20:35:07.041640 14595 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:07.042695 15936 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:07.043186 15936 raft_consensus.cc:686] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:06.950371 15918 raft_consensus.cc:686] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:07.044152 15179 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:07.044135 15936 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.045486 15918 consensus_queue.cc:237] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.051401 15692 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.002539 15821 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.013212 15953 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 20:35:07.004264 14549 raft_consensus.cc:2379] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:06.997517 15976 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
W20260806 20:35:07.007347 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 57485.6us
W20260806 20:35:07.007481 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 86918.2us
I20260806 20:35:07.009115 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.008250 16003 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.014375 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:06.958246 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.013s sys 0.043s
W20260806 20:35:07.089385 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 913538us
I20260806 20:35:07.012099 15915 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:07.094833 14442 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.019135 15842 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.028563 14624 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:07.035795 15183 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:07.020387 15908 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.083525 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.092526 14521 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:07.123971 14630 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:07.084417 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:07.090737 16001 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.125358 15153 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:07.125919 14595 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:07.114243 15723 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.116468 15868 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:07.128012 15953 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:07.136883 15944 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:07.137455 15944 raft_consensus.cc:686] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:07.139148 15837 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:07.139592 15837 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:07.139951 15837 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:07.140702 15811 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:07.141142 15811 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:07.141541 15811 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:07.116276 15842 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.144693 15860 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.147982 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.148756 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.150417 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:07.094703 15915 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:07.151180 14632 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:07.154420 14444 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.154610 14441 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:07.115444 15993 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:07.107913 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.113336 14475 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
W20260806 20:35:07.163044 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 51861us
I20260806 20:35:07.166006 14518 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.168015 14518 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.146880 15860 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.152566 15842 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:07.156651 15915 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:07.128342 15849 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260806 20:35:07.160774 15808 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:07.160306 15946 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:07.151154 16001 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:07.211848 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 43941.1us
I20260806 20:35:07.160619 15908 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.189949 15911 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:07.182391 15934 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:07.236721 15934 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.144721 15953 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.183374 15976 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.235054 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:07.214395 15882 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.243492 14518 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.247151 14518 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.229039 15842 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.251204 14518 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.245548 15911 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.254746 16002 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:07.276469 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 184022us
I20260806 20:35:07.240685 15908 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.281070 15186 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:07.283358 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 120793us
I20260806 20:35:07.268160 15882 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.239107 15944 consensus_queue.cc:237] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.280196 15743 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.301458 14595 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:07.281499 15911 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:07.260952 15842 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.245810 15976 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.280445 15692 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:07.314630 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 102198us
I20260806 20:35:07.309264 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:07.281213 16002 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.316830 15911 raft_consensus.cc:686] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:07.354616 15953 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:07.472071 15953 raft_consensus.cc:686] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:07.473328 14469 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:07.473264 15953 consensus_queue.cc:237] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.312350 15934 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:07.316223 15182 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:07.303470 15847 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.308899 15743 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.477622 15743 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.454747 15183 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:35:07.451881 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.374817 15185 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:07.320158 15692 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.462891 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.463768 14550 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:07.468676 14470 raft_consensus.cc:1261] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:07.471143 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.503741 14470 raft_consensus.cc:1261] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:07.461134 15914 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.480943 14479 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 20:35:07.482561 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 173408us
I20260806 20:35:07.508674 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:07.462663 16001 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.512131 14471 raft_consensus.cc:1261] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 20:35:07.517113 14471 pending_rounds.cc:76] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 1
I20260806 20:35:07.519209 14518 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:07.517460 14471 pending_rounds.cc:97] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20260806 20:35:07.528687 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 53284.1us
I20260806 20:35:07.504622 15979 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:07.530380 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 251160us
I20260806 20:35:07.481812 15743 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.552642 14631 raft_consensus.cc:1261] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:07.553058 15743 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.555613 16027 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.501353 16014 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:07.529356 14471 raft_consensus.cc:2866] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:35:07.508060 14654 debug-util.cc:398] Leaking SignalData structure 0x7b080008cf40 after lost signal to thread 13285
I20260806 20:35:07.533761 15979 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.568933 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.534403 15908 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.538723 15976 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.539659 16025 consensus_queue.cc:1046] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:35:07.543185 16023 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.559605 16017 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.531271 15882 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:07.545779 14469 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:07.595784 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:07.596803 14477 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:07.546303 15836 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:07.597743 15836 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.600150 16002 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:07.582544 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.605440 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.609848 16017 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.610754 16012 raft_consensus.cc:3032] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:07.611137 16012 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:07.612110 16006 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.579s user 0.001s sys 0.003s
I20260806 20:35:07.558593 14629 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:35:07.566797 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.478399 15847 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.571203 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:35:07.575644 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 70282.6us
I20260806 20:35:07.556432 15743 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.581279 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.622295 14632 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
W20260806 20:35:07.573187 15962 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.626s user 0.013s sys 0.000s
I20260806 20:35:07.543669 15853 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:07.591701 14470 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 20:35:07.555665 14631 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:07.546312 14624 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:07.611915 16023 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:07.653803 15837 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.513s user 0.012s sys 0.000s
I20260806 20:35:07.654376 16023 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.592989 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:07.654620 15837 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.516119 15914 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.660945 15837 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.661345 15914 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:07.549659 15811 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.663041 15692 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.664261 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.665478 15811 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.666024 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.666534 15184 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:07.624989 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.631479 15153 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:07.633682 15825 consensus_queue.cc:1035] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:07.634320 15853 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:07.634796 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.636375 15930 raft_consensus.cc:3032] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:07.639393 15808 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:07.685662 14630 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:07.641248 16007 log.cc:924] Time spent T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.607s user 0.003s sys 0.000s
I20260806 20:35:07.641160 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.694806 14628 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:07.709729 14625 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:07.642513 14596 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:07.640517 15836 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:07.630633 16000 consensus_meta.cc:287] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.592s user 0.009s sys 0.005s
I20260806 20:35:07.643167 15879 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.644521 14595 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:07.716223 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 71866us
I20260806 20:35:07.645001 15847 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:07.645826 15946 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:07.618295 16011 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20260806 20:35:07.644559 15821 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.647604 16024 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:07.616593 15882 raft_consensus.cc:686] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:07.627192 14654 debug-util.cc:398] Leaking SignalData structure 0x7b08000e0ac0 after lost signal to thread 14407
I20260806 20:35:07.612869 15842 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.566249 16027 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.527501 15911 consensus_queue.cc:237] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.650075 16031 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.650697 15915 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.655490 14471 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:07.656986 14518 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:07.753031 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 96143.9us
I20260806 20:35:07.676942 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.756785 14632 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
W20260806 20:35:07.628963 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48088.1us
I20260806 20:35:07.758226 14442 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.758260 15153 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:07.640389 16017 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.678020 15930 raft_consensus.cc:729] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:07.672864 14629 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:35:07.713370 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 70952.7us
I20260806 20:35:07.714213 15962 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:07.721300 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:07.727751 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:07.767310 14625 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:07.710667 15811 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:07.727048 16028 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.765236 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.736254 15808 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.778990 15184 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 20:35:07.781167 14595 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:07.726436 15879 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.752920 15944 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 20:35:07.797849 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.802464 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.803339 14554 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:07.761912 16026 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:07.726243 16000 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.737433 15821 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.776713 16017 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.811937 14625 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:07.817965 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.806828 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.731398 15946 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.827504 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.785108 16020 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:07.792788 15996 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:07.807155 14654 debug-util.cc:398] Leaking SignalData structure 0x7b08003a2900 after lost signal to thread 14503
I20260806 20:35:07.844139 15153 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:07.822468 14552 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:07.798458 15944 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260806 20:35:07.839615 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:07.769992 15945 consensus_queue.cc:1046] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260806 20:35:07.770146 16014 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260806 20:35:07.845899 15962 raft_consensus.cc:686] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:07.781765 15934 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:07.816833 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 500956bd1a3846a3a7b16c51ddecbc44 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:07.710523 16012 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:07.885792 14356 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 155077us
I20260806 20:35:07.847252 14554 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:07.785948 15811 raft_consensus.cc:686] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:07.809077 15914 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.864275 14627 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:07.803908 16043 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 20:35:07.866377 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 83758us
I20260806 20:35:07.866370 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:07.866915 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.868362 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.869351 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:07.874009 14632 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:35:07.786158 15808 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:07.883989 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 79211.5us
I20260806 20:35:07.899073 16052 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260806 20:35:07.913429 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.852952 16032 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:07.760963 16031 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.851686 16017 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.853695 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:07.877502 15842 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:07.910686 14442 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:07.920831 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:07.941818 14629 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:07.820273 15882 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.862015 16020 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.927451 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:07.781008 16028 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:07.933030 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 49387.8us
I20260806 20:35:07.937994 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:07.937170 14654 debug-util.cc:398] Leaking SignalData structure 0x7b080039f280 after lost signal to thread 14581
I20260806 20:35:07.955931 14550 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:07.925473 14444 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:07.926776 15962 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.906275 16043 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:07.966948 16031 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.967221 16043 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:07.969364 16027 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:07.927733 15811 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.959165 16036 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.959496 14474 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:07.948588 14469 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:07.960377 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 49790.4us
I20260806 20:35:07.905869 15934 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:07.961686 14633 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:07.961109 16051 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:07.964259 15879 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:07.974169 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:07.959968 16037 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:07.988102 14548 raft_consensus.cc:2379] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:07.955030 15692 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:07.943030 16034 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:07.986799 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:07.978554 16015 log.cc:924] Time spent T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.563s user 0.002s sys 0.000s
I20260806 20:35:07.910086 15914 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:07.961135 15821 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:08.010149 15153 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:07.980172 15915 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.011752 15151 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:08.049181 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 62463us
I20260806 20:35:08.055799 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:07.989481 16051 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:08.065167 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 53507.8us
W20260806 20:35:08.041406 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 43565.7us
I20260806 20:35:08.082535 14444 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:08.083107 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.083916 14555 raft_consensus.cc:2379] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:08.087817 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.001925 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:08.093886 15151 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:08.093871 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.047294 16003 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:08.048348 15961 raft_consensus.cc:3032] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
W20260806 20:35:08.066641 14654 debug-util.cc:398] Leaking SignalData structure 0x7b08001f65c0 after lost signal to thread 14657
I20260806 20:35:08.099949 14550 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:08.060518 16037 raft_consensus.cc:686] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:07.934589 16032 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.110553 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.105423 14629 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:08.081110 16001 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:08.091729 16057 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:08.081674 16023 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:08.081391 16020 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.086810 15936 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.082494 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.095672 15879 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:08.074220 16051 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:08.089243 15915 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.095405 16000 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.103822 15821 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:08.109586 14654 debug-util.cc:398] Leaking SignalData structure 0x7b08001f65e0 after lost signal to thread 14658
I20260806 20:35:08.147045 14442 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:08.101830 16003 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:08.104566 15961 raft_consensus.cc:729] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:07.983110 16036 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.124639 15808 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:08.064759 16048 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.267s
I20260806 20:35:08.158792 16001 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 20:35:08.174892 16010 log.cc:924] Time spent T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.746s user 0.005s sys 0.000s
I20260806 20:35:08.177525 16057 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:08.153460 15178 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:08.228978 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 111) took 1144 ms. Trace:
W20260806 20:35:08.179078 16038 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.305s user 0.000s sys 0.002s
I20260806 20:35:08.180187 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:08.181035 15934 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.159566 15915 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.163623 15821 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:08.208671 15996 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.150377 15936 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.183010 14473 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
W20260806 20:35:08.184017 14654 debug-util.cc:398] Leaking SignalData structure 0x7b08002c3220 after lost signal to thread 15212
I20260806 20:35:08.184235 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:08.154017 15879 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:08.168700 16042 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.253s user 0.000s sys 0.002s
W20260806 20:35:08.210428 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 116610us
I20260806 20:35:08.029028 16034 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:08.165252 16003 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:08.229453 14626 rpcz_store.cc:276] 0806 20:35:07.084769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.084879 (+ 110us) service_pool.cc:227] Handling call
0806 20:35:07.115537 (+ 30658us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:07.125167 (+ 9630us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:07.125217 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:07.969646 (+844429us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b2c000b26a8. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:07.970191 (+ 545us) log.cc:842] Serialized 26 byte log entry
0806 20:35:08.048468 (+ 78277us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:08.048510 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:08.048600 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:08.190123 (+141523us) raft_consensus.cc:1671] finished
0806 20:35:08.190157 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:08.228893 (+ 38736us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5609,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":22071,"spinlock_wait_cycles":6713728,"thread_start_us":77629,"threads_started":1,"wal-append.queue_time_us":131393}
W20260806 20:35:08.235574 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 134503us
I20260806 20:35:08.229110 16017 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:08.439136 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:08.440830 15853 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.441615 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:08.441817 15853 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:08.442871 14473 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:08.173334 14631 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:08.210734 16001 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:08.206599 16052 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:08.255554 16036 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:08.256108 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.459421 14554 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:08.228679 16057 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:08.268262 15918 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.998s user 0.020s sys 0.024s
W20260806 20:35:08.270135 14654 debug-util.cc:398] Leaking SignalData structure 0x7b080032c060 after lost signal to thread 15213
I20260806 20:35:08.300261 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:08.259254 15821 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:08.263393 15934 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.168100 16058 consensus_queue.cc:1046] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.237s
I20260806 20:35:08.258095 15914 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:08.485702 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.034s sys 0.051s
W20260806 20:35:08.486234 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.034s sys 0.051s
I20260806 20:35:08.469735 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:08.280567 15879 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:08.473891 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 173802us
I20260806 20:35:08.457329 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:08.468190 15918 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.204s user 0.020s sys 0.024s
I20260806 20:35:08.473008 14474 raft_consensus.cc:2379] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 2: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:08.528333 15808 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:08.531690 15808 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:08.534576 15808 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:08.536247 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:08.264039 15996 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.462538 16001 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:08.468626 15847 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.490319 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:08.479435 16036 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:08.490044 16046 log.cc:924] Time spent T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.269s user 0.001s sys 0.001s
I20260806 20:35:08.462703 15993 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:08.528594 15868 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:08.508986 14471 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.013s sys 0.073s
W20260806 20:35:08.571072 14471 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.743s user 0.014s sys 0.073s
I20260806 20:35:08.474308 16028 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:08.483458 16039 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.266606 15961 consensus_queue.cc:260] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:08.551409 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 61217.5us
I20260806 20:35:08.487212 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:08.471506 16057 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:08.526899 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:08.242939 16037 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:08.553216 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1415) took 1086 ms. Trace:
I20260806 20:35:08.588719 15853 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:08.489070 15934 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.584852 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.568408 16032 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.472743 16052 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:08.605389 15918 consensus_queue.cc:1046] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
W20260806 20:35:08.610056 16066 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20260806 20:35:08.610116 16028 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:08.584473 16056 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.439s user 0.002s sys 0.000s
I20260806 20:35:08.610395 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1423) took 1107 ms. Trace:
I20260806 20:35:08.639351 14476 rpcz_store.cc:276] 0806 20:35:07.502791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.505983 (+ 3192us) service_pool.cc:227] Handling call
0806 20:35:07.947161 (+441178us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:07.947413 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:08.000405 (+ 52992us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:08.000426 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:08.000518 (+ 92us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:08.000547 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:08.610150 (+609603us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b5400332cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:08.610201 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:08.610338 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124998272}
I20260806 20:35:08.585857 15868 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.634634 16048 raft_consensus.cc:3032] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:08.578940 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 363) took 1634 ms. Trace:
I20260806 20:35:08.562736 15847 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.642402 16028 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.666468 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.671249 15953 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:08.691448 16028 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:08.632853 14474 raft_consensus.cc:1261] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:08.646158 15915 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.660069 15180 rpcz_store.cc:276] 0806 20:35:06.944709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:06.944836 (+ 127us) service_pool.cc:227] Handling call
0806 20:35:06.945465 (+ 629us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:06.946392 (+ 927us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:06.946471 (+ 79us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:06.947290 (+ 819us) log.cc:842] Serialized 26 byte log entry
0806 20:35:07.239790 (+292500us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:07.283855 (+ 44065us) log.cc:842] Serialized 14 byte log entry
0806 20:35:07.303336 (+ 19481us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:07.308822 (+ 5486us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:07.598599 (+289777us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b1800403120. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb764 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:08.114130 (+515531us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:08.553740 (+439610us) raft_consensus.cc:1671] finished
0806 20:35:08.555328 (+ 1588us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:08.572776 (+ 17448us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7794,"prepare.queue_time_us":27782,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":18258,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":11365,"spinlock_wait_cycles":24554112,"thread_start_us":263003,"threads_started":1,"wal-append.queue_time_us":339746}
I20260806 20:35:08.655138 16048 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:08.591114 16039 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.692335 15930 consensus_queue.cc:260] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:08.756930 14368 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45970 (request call id 232) took 3548 ms. Trace:
I20260806 20:35:08.728546 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.804630 14554 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:08.720786 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.806229 14549 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:08.756917 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.835582 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:08.803424 14368 rpcz_store.cc:276] 0806 20:35:05.208897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.209783 (+ 886us) service_pool.cc:227] Handling call
0806 20:35:08.756850 (+3547067us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:07.232823 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:07.237146 (+ 4323us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:07.237190 (+ 44us) write_op.cc:435] Acquired schema lock
0806 20:35:07.237209 (+ 19us) tablet.cc:660] Decoding operations
0806 20:35:07.237650 (+ 441us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:07.237743 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:07.237775 (+ 32us) tablet.cc:683] Acquiring locks for 2 operations
0806 20:35:07.238104 (+ 329us) tablet.cc:699] Row locks acquired
0806 20:35:07.238125 (+ 21us) write_op.cc:260] PREPARE: finished
0806 20:35:07.238277 (+ 152us) write_op.cc:270] Start()
0806 20:35:07.238389 (+ 112us) write_op.cc:276] Timestamp: P: 1786048507238232 usec, L: 0
0806 20:35:07.238410 (+ 21us) op_driver.cc:331] REPLICATION: starting
0806 20:35:07.245550 (+ 7140us) log.cc:842] Serialized 925 byte log entry
0806 20:35:07.642303 (+396753us) op_driver.cc:444] REPLICATION: finished
0806 20:35:07.842531 (+200228us) write_op.cc:301] APPLY: starting
0806 20:35:07.850414 (+ 7883us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:07.877963 (+ 27549us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:07.895842 (+ 17879us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:07.989467 (+ 93625us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:08.052576 (+ 63109us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 20:35:08.065221 (+ 12645us) write_op.cc:312] APPLY: finished
0806 20:35:08.098648 (+ 33427us) log.cc:842] Serialized 29 byte log entry
0806 20:35:08.112886 (+ 14238us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:08.119280 (+ 6394us) write_op.cc:454] Released schema lock
0806 20:35:08.126566 (+ 7286us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:08.173580 (+ 47014us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:08.200544 (+ 26964us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7791,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":150619,"apply.run_cpu_time_us":4935,"apply.run_wall_time_us":405706,"num_ops":2,"prepare.queue_time_us":364190,"prepare.run_cpu_time_us":3503,"prepare.run_wall_time_us":44816,"replication_time_us":403784,"thread_start_us":390947,"threads_started":2}]]}
W20260806 20:35:08.824339 14469 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.826s user 0.016s sys 0.007s
I20260806 20:35:08.843461 16029 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:08.848583 14469 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:08.851629 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.697355 16012 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:08.672271 15953 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:08.849730 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1431) took 1046 ms. Trace:
I20260806 20:35:08.718266 15868 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.718803 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:08.726583 15996 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.717752 16070 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.860749 14596 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:08.904572 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 43935.2us
I20260806 20:35:08.762077 15915 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:08.762889 15847 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:08.796919 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.885022 14478 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
W20260806 20:35:08.693181 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 127984us
I20260806 20:35:08.834973 16031 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.896088 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.920516 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1326) took 1010 ms. Trace:
I20260806 20:35:08.845028 15930 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:08.920902 14470 rpcz_store.cc:276] 0806 20:35:07.910001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.910157 (+ 156us) service_pool.cc:227] Handling call
0806 20:35:07.939070 (+ 28913us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:07.957252 (+ 18182us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:07.957301 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:07.986326 (+ 29025us) log.cc:842] Serialized 26 byte log entry
0806 20:35:08.259691 (+273365us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:08.259759 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:08.259848 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:08.920282 (+660434us) raft_consensus.cc:1671] finished
0806 20:35:08.920304 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:08.920435 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33736,"raft.queue_time_us":18008,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":12466,"thread_start_us":273051,"threads_started":1,"wal-append.queue_time_us":487469}
I20260806 20:35:08.850482 16017 raft_consensus.cc:2741] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:08.902220 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:08.728641 16048 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:08.902226 16068 log.cc:924] Time spent T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20260806 20:35:08.874778 14469 rpcz_store.cc:276] 0806 20:35:07.803161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.807213 (+ 4052us) service_pool.cc:227] Handling call
0806 20:35:08.849638 (+1042425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:08.913497 14596 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:08.914129 15915 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:08.915169 15868 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:08.865062 16030 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:08.696496 15979 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:08.843739 16013 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.927990 14547 raft_consensus.cc:2379] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:08.984607 14549 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:08.992640 14549 raft_consensus.cc:595] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:08.984856 15953 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:08.991540 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:08.849354 16029 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:08.882369 16020 consensus_queue.cc:1159] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:35:08.997666 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.004767 16040 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:08.987497 16044 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:09.012300 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 500956bd1a3846a3a7b16c51ddecbc44 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:08.990070 15847 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:08.948405 15868 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:08.947536 16070 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.056500 16052 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:09.058327 16044 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:09.059105 16044 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.056883 15946 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:08.939314 15934 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:09.075167 14552 raft_consensus.cc:3032] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:08.929049 16031 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.005631 16040 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.075853 14552 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:09.091307 16019 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:09.073158 16043 consensus_meta.cc:287] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.105s user 0.015s sys 0.003s
I20260806 20:35:09.030674 15847 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:09.076792 15908 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:09.098147 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:08.993245 16034 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:09.062315 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:09.068070 15722 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:09.019055 16039 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.108827 14478 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:09.053210 15868 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.014112 16030 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.106302 14552 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.121167 14552 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:09.086467 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.125007 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.077258 15946 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.127369 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.128196 14470 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:09.098937 15743 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:09.113823 15847 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:08.595386 14475 rpcz_store.cc:276] 0806 20:35:07.462102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.470855 (+ 8753us) service_pool.cc:227] Handling call
0806 20:35:07.471262 (+ 407us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:07.478631 (+ 7369us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:07.478663 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:07.479146 (+ 483us) log.cc:842] Serialized 26 byte log entry
0806 20:35:07.713388 (+234242us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:07.876601 (+163213us) log.cc:842] Serialized 14 byte log entry
0806 20:35:07.876827 (+ 226us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:07.876935 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:08.512250 (+635315us) raft_consensus.cc:1671] finished
0806 20:35:08.512294 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:08.548638 (+ 36344us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87641,"prepare.queue_time_us":172265,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":152189,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":22887,"thread_start_us":396414,"threads_started":2,"wal-append.queue_time_us":298580}
I20260806 20:35:09.136626 16032 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:09.122056 15722 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.141101 16032 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.144145 15722 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:09.114991 15908 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.154142 16029 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.008152 16013 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.158483 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.159855 14555 raft_consensus.cc:2379] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:09.159895 15911 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.979s user 0.031s sys 0.155s
I20260806 20:35:09.167783 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.112574 16043 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.175992 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:09.137087 14546 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:09.168588 14521 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.110s user 0.043s sys 0.119s
W20260806 20:35:09.178056 14521 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.120s user 0.043s sys 0.119s
I20260806 20:35:09.172685 16027 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.174779 16030 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:09.179741 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:09.135883 16019 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:09.181898 15821 consensus_meta.cc:287] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.692s user 0.009s sys 0.017s
I20260806 20:35:09.183413 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.207159 14546 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:09.163753 15915 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:09.185842 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:09.186085 15185 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:35:09.210562 16057 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.604s user 0.009s sys 0.009s
I20260806 20:35:09.211807 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.201006 14555 raft_consensus.cc:2379] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:09.203459 15152 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.034s sys 0.116s
W20260806 20:35:09.206359 15879 consensus_meta.cc:287] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.696s user 0.011s sys 0.012s
I20260806 20:35:09.222611 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.134883 15946 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.198998 14442 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:09.218839 14548 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:09.227365 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:09.179828 15911 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.999s user 0.031s sys 0.156s
W20260806 20:35:09.222290 15152 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.108s user 0.035s sys 0.116s
I20260806 20:35:09.149755 16070 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.139119 15743 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.185860 15936 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:09.223017 15879 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:09.212317 16003 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.754s user 0.003s sys 0.011s
I20260806 20:35:09.213723 15908 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.229667 14442 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:09.280207 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 50642.3us
W20260806 20:35:09.281354 16075 log.cc:924] Time spent T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20260806 20:35:09.187296 16027 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.258891 16023 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:09.259023 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.228197 14555 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:09.246266 14521 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:09.305461 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 59314.7us
I20260806 20:35:09.187971 15948 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:09.271844 15879 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.286389 14552 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:09.287431 16070 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:09.201244 15150 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:09.291590 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:09.284219 16013 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.304344 16023 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:09.315922 16023 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:09.316211 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.292836 15936 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:09.313000 15957 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:09.317708 15936 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.318857 15957 raft_consensus.cc:686] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:09.297401 15915 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:09.320204 15946 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.320614 15948 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.320870 15915 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.272708 16003 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.325227 16043 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.307268 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:09.327767 16001 consensus_meta.cc:287] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.765s user 0.007s sys 0.017s
I20260806 20:35:09.327586 15182 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:09.305557 14596 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:09.330843 16027 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.255941 16019 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:09.228586 16032 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:09.310815 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.357321 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:09.336297 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 158438us
I20260806 20:35:09.344956 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.346159 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.347826 14479 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 20:35:09.358374 14477 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:09.348606 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.349802 16019 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:09.351941 15934 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:09.366148 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.243763 16072 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.357468 14555 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:09.371636 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:09.372372 14555 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.309681 16017 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:09.328037 16003 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.364471 14441 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:09.347501 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.367002 14595 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:09.216084 15821 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.379689 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.366977 14552 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:09.382639 14441 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:09.363786 15860 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:09.394012 14442 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:09.368466 15934 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:09.394903 15934 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.398651 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.410748 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.221312 16057 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.378319 15722 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:09.412492 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.426910 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.419780 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.422698 16018 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:09.386610 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.844s user 0.027s sys 0.076s
I20260806 20:35:09.429414 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:09.431806 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.889s user 0.027s sys 0.076s
W20260806 20:35:09.433631 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 890881us
I20260806 20:35:09.386732 16013 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:09.433631 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.436945 14477 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:09.440322 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.378091 15996 consensus_queue.cc:1035] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260806 20:35:09.417188 15908 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:09.392014 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.416144 15946 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:09.380087 16032 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260806 20:35:09.465057 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.468196 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.447050 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.426041 15915 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:09.392588 16072 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.333503 16001 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.417446 16070 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:09.380381 16019 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.389436 16040 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.332532 15957 consensus_queue.cc:237] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.459909 15184 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:35:09.383777 16017 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.455914 15860 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:09.461854 14474 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:09.456910 16043 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:09.483525 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.506031 14477 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:09.463024 15936 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:09.484683 15911 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.484565 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.434867 16031 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:09.435781 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 53987us
I20260806 20:35:09.448989 16018 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:09.458813 15908 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:09.464804 15946 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:09.507053 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.483247 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:09.522936 14478 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
W20260806 20:35:09.501004 16051 consensus_meta.cc:287] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.338s user 0.004s sys 0.021s
W20260806 20:35:09.534160 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.922s user 0.089s sys 0.124s
I20260806 20:35:09.499559 15915 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:09.545580 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.087s user 0.089s sys 0.125s
I20260806 20:35:09.557890 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:09.566567 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.522284 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:09.516631 15948 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:09.567371 14477 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:09.455276 16013 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:09.538585 15930 consensus_meta.cc:287] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.602s user 0.004s sys 0.013s
I20260806 20:35:09.581398 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.532419 15936 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:09.542225 16018 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.590509 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:09.546058 15908 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.586979 15178 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:09.599062 16036 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.544955 15743 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.605651 14551 raft_consensus.cc:1261] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:09.504639 15860 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.500626 16070 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.546046 15946 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.665022 15821 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.575323 15183 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:09.615449 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:09.583277 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:09.669653 16036 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.585527 15852 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.579694 15915 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.535627 15911 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:09.532366 16043 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.574941 16051 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:09.906250 14598 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.335s user 0.043s sys 0.060s
W20260806 20:35:09.905324 14471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.262s user 0.037s sys 0.073s
W20260806 20:35:10.027248 14471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.384s user 0.057s sys 0.102s
I20260806 20:35:09.588122 15948 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:10.059579 14598 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.596s user 0.048s sys 0.089s
W20260806 20:35:09.913471 14554 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.021s sys 0.062s
W20260806 20:35:10.075995 14554 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.889s user 0.040s sys 0.086s
I20260806 20:35:09.855293 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.679625 14555 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
W20260806 20:35:09.834478 16044 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.775s user 0.008s sys 0.012s
W20260806 20:35:09.751363 14469 consensus_meta.cc:287] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.528s user 0.013s sys 0.002s
W20260806 20:35:09.757704 16012 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.883s user 0.012s sys 0.005s
I20260806 20:35:09.783519 16031 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.671661 16057 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.792383 16072 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:10.106840 16069 log.cc:924] Time spent T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.204s user 0.000s sys 0.000s
I20260806 20:35:10.108006 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:10.108327 16036 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:09.892282 15847 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.740s user 0.009s sys 0.005s
I20260806 20:35:10.121860 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:09.671631 15983 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:09.680097 16052 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:09.894583 16019 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:09.867535 16001 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.079912 15948 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.076846 14546 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:35:09.776708 16032 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:10.062074 15743 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:09.754577 16013 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.098039 16073 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:09.853971 16017 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.102666 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:09.678382 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 102308us
I20260806 20:35:09.610647 15936 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:09.941306 16029 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:10.078938 16000 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.206s user 0.014s sys 0.018s
W20260806 20:35:10.175276 16000 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.307s user 0.015s sys 0.019s
I20260806 20:35:10.175382 15948 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.175974 15948 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:10.176111 16018 consensus_meta.cc:287] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.503s user 0.009s sys 0.000s
I20260806 20:35:10.118211 14549 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:10.126873 14553 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 4. (term mismatch)
I20260806 20:35:10.176712 16018 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:10.177336 16023 consensus_meta.cc:287] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.861s user 0.009s sys 0.004s
I20260806 20:35:10.177434 14553 pending_rounds.cc:76] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 3
I20260806 20:35:10.124866 16031 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.177800 16023 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.179067 14477 raft_consensus.cc:2379] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:10.179648 16031 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.151093 14473 raft_consensus.cc:2354] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3.
I20260806 20:35:10.120868 16072 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:10.152820 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:10.179803 16073 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.180737 15183 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:10.180536 16072 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.154289 16020 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:10.076521 15852 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.177810 14553 pending_rounds.cc:97] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.4
I20260806 20:35:10.182127 16073 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.182154 14553 raft_consensus.cc:2894] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Aborting config change with OpId 1.4: Aborted: Op aborted by new leader
I20260806 20:35:10.182691 16073 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:10.182523 16072 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.183099 14553 raft_consensus.cc:2866] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:10.183368 15852 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.162736 14625 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 20:35:10.174413 16029 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.172034 16053 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:09.868068 15930 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:10.094883 16070 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.181752 16020 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.184634 16053 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.185465 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:10.186718 16029 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.186829 16020 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.179601 16018 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.187633 14626 raft_consensus.cc:1261] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:10.188113 16053 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.184715 15930 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.189137 15948 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.191421 15930 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.193032 15948 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.193490 15911 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.127386 15847 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.195753 16051 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.197467 15847 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.200141 15915 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.131845 16052 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.132305 16076 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:10.209625 16076 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.214061 16043 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:10.217713 15934 consensus_meta.cc:287] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.822s user 0.007s sys 0.007s
I20260806 20:35:10.218225 15934 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:10.225729 15868 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.090s user 0.007s sys 0.007s
I20260806 20:35:10.226435 15868 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.231238 15936 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.234375 15150 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:10.183228 16073 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.183741 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54486 (request call id 1192) took 3479 ms. Trace:
I20260806 20:35:10.100639 14469 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:10.183421 16072 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:10.258663 16072 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.260826 16051 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:10.262183 16072 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.261862 16051 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.265362 16028 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:10.254949 14442 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:10.266804 16051 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.267532 16051 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:10.267926 16051 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:10.268512 16051 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:10.269361 16031 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:10.256073 14553 rpcz_store.cc:276] 0806 20:35:06.704675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:06.704822 (+ 147us) service_pool.cc:227] Handling call
0806 20:35:06.803905 (+ 99083us) log.cc:842] Serialized 508 byte log entry
0806 20:35:07.457622 (+653717us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b1800302ce0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:07.611155 (+153533us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:08.650834 (+1039679us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54000783a0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b3481771 00007fb9b348c220 00007fb9b348c1c8 00007fb9b348c148 00007fb9b348c103 00007fb9b348b38c 00007fb9b94fc206 00007fb9b94f4808 00007fb9b34c838b 00007fb9b34cae80
0806 20:35:10.183650 (+1532816us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31497,"raft.queue_time_us":21919,"raft.run_cpu_time_us":1020,"raft.run_wall_time_us":145887,"spinlock_wait_cycles":804446336,"thread_start_us":122853,"threads_started":1,"wal-append.queue_time_us":228869,"wal-append.run_cpu_time_us":46616,"wal-append.run_wall_time_us":2728715}
I20260806 20:35:10.269888 16031 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.270920 16067 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:10.271984 16031 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.271497 16067 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.257458 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:10.281359 14549 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:10.280290 14476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.572s user 0.011s sys 0.020s
W20260806 20:35:10.284516 14476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.576s user 0.011s sys 0.020s
I20260806 20:35:10.263447 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:10.294178 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:10.294910 14555 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:10.282212 16043 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:10.166707 15743 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.134374 15983 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.166858 16032 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:10.287889 16031 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:10.302641 14470 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.166s user 0.082s sys 0.143s
W20260806 20:35:10.304612 14470 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.168s user 0.082s sys 0.143s
I20260806 20:35:10.303435 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:10.307255 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:10.310338 15743 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.293635 14547 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:10.316108 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1401) took 1373 ms. Trace:
I20260806 20:35:10.316499 14548 rpcz_store.cc:276] 0806 20:35:08.942451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:08.942623 (+ 172us) service_pool.cc:227] Handling call
0806 20:35:10.315923 (+1373300us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b54000a8054. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:10.316035 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":506883456}
I20260806 20:35:10.291796 16017 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:10.298712 16043 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:10.292526 16023 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.289379 15150 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:10.323999 15946 consensus_meta.cc:287] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.551s user 0.013s sys 0.000s
I20260806 20:35:10.323319 16043 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:10.318956 16031 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.101717 16044 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:10.326812 15811 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.779s user 0.017s sys 0.015s
I20260806 20:35:10.323201 16017 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:10.349982 15962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.007s sys 0.024s
I20260806 20:35:10.326086 16051 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:10.349346 15722 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.891s user 0.014s sys 0.017s
I20260806 20:35:10.346453 16003 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:10.350426 16076 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.342746 15934 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.338953 15946 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:10.360684 15811 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.823s user 0.017s sys 0.015s
W20260806 20:35:10.353235 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 69755.9us
W20260806 20:35:10.366034 15962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.925s user 0.008s sys 0.024s
W20260806 20:35:10.367671 15722 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.917s user 0.014s sys 0.017s
I20260806 20:35:10.367440 16003 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:10.367630 15868 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:10.447539 15908 consensus_meta.cc:287] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.766s user 0.012s sys 0.001s
W20260806 20:35:10.370967 16037 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.481s user 0.010s sys 0.024s
I20260806 20:35:10.452493 16003 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:10.455067 16040 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.910s user 0.016s sys 0.018s
I20260806 20:35:10.452706 15722 raft_consensus.cc:686] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:35:10.455214 16058 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.824s user 0.008s sys 0.029s
W20260806 20:35:10.464710 16037 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.580s user 0.011s sys 0.024s
W20260806 20:35:10.680258 16040 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.136s user 0.016s sys 0.020s
W20260806 20:35:10.690968 16058 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.078s user 0.008s sys 0.029s
I20260806 20:35:10.839339 16073 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.457677 15908 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:10.690342 15983 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:10.747100 16032 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:10.909315 16044 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.909776 16031 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:10.869242 16049 consensus_queue.cc:1046] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.135s
W20260806 20:35:11.127475 15957 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.007s sys 0.012s
I20260806 20:35:11.035638 16017 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:11.221295 16084 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.347s user 0.000s sys 0.000s
W20260806 20:35:11.223289 16085 log.cc:924] Time spent T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.334s user 0.000s sys 0.000s
W20260806 20:35:11.584158 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 7f41753baced49a1997940dfc28b2e00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Aborted: Op aborted by new leader
W20260806 20:35:12.269011 14404 debug-util.cc:398] Leaking SignalData structure 0x7b080003b520 after lost signal to thread 13282
W20260806 20:35:12.275483 16098 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.875s user 0.000s sys 0.001s
W20260806 20:35:12.305883 14404 debug-util.cc:398] Leaking SignalData structure 0x7b080003b540 after lost signal to thread 13285
W20260806 20:35:12.492146 16106 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.744s user 0.000s sys 0.001s
I20260806 20:35:12.973908 15961 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:12.979387 16107 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:12.980055 16107 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:12.982676 16107 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:12.983343 16107 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:12.983855 16107 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:12.986129 16107 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:12.986625 16107 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:12.988368 16107 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:12.991184 16107 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:12.991842 15946 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:12.994305 14477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1449) took 3481 ms (client timeout 2990 ms). Trace:
W20260806 20:35:12.994622 16003 consensus_meta.cc:287] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 2.305s user 0.019s sys 0.012s
W20260806 20:35:12.994668 14477 rpcz_store.cc:269] 0806 20:35:09.512595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.590205 (+ 77610us) service_pool.cc:227] Handling call
0806 20:35:12.994106 (+3403901us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b54001a54d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:12.994222 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1316715776}
I20260806 20:35:12.995252 16003 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:12.996531 14477 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:12.997550 16003 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:12.997202 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1338) took 3653 ms. Trace:
I20260806 20:35:12.998304 14477 rpcz_store.cc:276] 0806 20:35:09.343892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:12.995984 (+3652092us) service_pool.cc:227] Handling call
0806 20:35:12.996337 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:12.996932 (+ 595us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:12.997147 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:13.006091 14555 consensus_meta.cc:287] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 2.711s user 0.031s sys 0.012s
I20260806 20:35:13.006673 14555 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:13.007333 14555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1223) took 2749 ms (client timeout 1549 ms). Trace:
W20260806 20:35:13.007767 14555 rpcz_store.cc:269] 0806 20:35:10.257449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:10.285454 (+ 28005us) service_pool.cc:227] Handling call
0806 20:35:13.007244 (+2721790us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.006871 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1434) took 3553 ms. Trace:
W20260806 20:35:13.009009 14546 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 2.835s user 0.026s sys 0.030s
I20260806 20:35:13.010274 14546 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:13.010819 14546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1209) took 3454 ms (client timeout 2069 ms). Trace:
W20260806 20:35:13.011149 14546 rpcz_store.cc:269] 0806 20:35:09.556897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.576870 (+ 19973us) service_pool.cc:227] Handling call
0806 20:35:13.010758 (+3433888us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:13.011896 15150 connection.cc:232] RPC call timeout handler was delayed by -2.2263983559969347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
W20260806 20:35:13.012185 14444 connection.cc:232] RPC call timeout handler was delayed by -2.6964668099826667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260806 20:35:13.013243 14444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.772s (SENT) [suppressed 3 similar messages]
I20260806 20:35:13.015086 16107 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:11.185679 15957 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s user 0.008s sys 0.013s
I20260806 20:35:13.015792 16107 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.019636 16107 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.009029 14477 rpcz_store.cc:276] 0806 20:35:09.453849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:12.999119 (+3545270us) service_pool.cc:227] Handling call
0806 20:35:12.999446 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:12.999761 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:12.999789 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:13.000189 (+ 400us) log.cc:842] Serialized 26 byte log entry
0806 20:35:13.001172 (+ 983us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:13.001211 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.001278 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:13.006651 (+ 5373us) raft_consensus.cc:1671] finished
0806 20:35:13.006672 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.006813 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":654,"threads_started":1,"wal-append.queue_time_us":1073}
I20260806 20:35:13.024287 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1435) took 3570 ms. Trace:
I20260806 20:35:13.026216 14477 rpcz_store.cc:276] 0806 20:35:09.454265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.023454 (+3569189us) service_pool.cc:227] Handling call
0806 20:35:13.023791 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.023963 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.023991 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.024032 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:13.024067 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.024119 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.024231 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.029157 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1436) took 3574 ms. Trace:
I20260806 20:35:13.030895 14477 rpcz_store.cc:276] 0806 20:35:09.454590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.028314 (+3573724us) service_pool.cc:227] Handling call
0806 20:35:13.028681 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.028848 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.028875 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.028915 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.028943 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.028994 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.029104 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.033509 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1437) took 3578 ms. Trace:
I20260806 20:35:13.022008 16067 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.031988 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1491) took 2868 ms. Trace:
I20260806 20:35:13.035867 14630 rpcz_store.cc:276] 0806 20:35:10.163482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:10.163728 (+ 246us) service_pool.cc:227] Handling call
0806 20:35:10.164100 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:10.248895 (+ 84795us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:10.248945 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:10.249138 (+ 193us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:10.249197 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.031608 (+2782411us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b5400303754. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:13.031699 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.031873 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":194322688}
I20260806 20:35:13.037796 14477 rpcz_store.cc:276] 0806 20:35:09.455324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.032729 (+3577405us) service_pool.cc:227] Handling call
0806 20:35:13.033046 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.033213 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.033239 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.033279 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.033309 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.033358 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.033460 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.040769 14477 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260806 20:35:13.041096 14547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1225) took 2727 ms (client timeout 2639 ms). Trace:
W20260806 20:35:13.042317 14547 rpcz_store.cc:269] 0806 20:35:10.313401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.040854 (+2727453us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:13.040996 (+ 142us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:35:13.042798 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1438) took 3585 ms. Trace:
I20260806 20:35:13.043797 14477 rpcz_store.cc:276] 0806 20:35:09.457575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.039644 (+3582069us) service_pool.cc:227] Handling call
0806 20:35:13.040382 (+ 738us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.042040 (+ 1658us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.042740 (+ 700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.046093 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1439) took 3582 ms. Trace:
I20260806 20:35:13.047994 14477 rpcz_store.cc:276] 0806 20:35:09.463351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.045253 (+3581902us) service_pool.cc:227] Handling call
0806 20:35:13.045589 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.045768 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.045795 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.045842 (+ 47us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.045876 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.045930 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.046041 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:13.048939 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1334) took 3706 ms (client timeout 2121 ms). Trace:
W20260806 20:35:13.049844 14474 rpcz_store.cc:269] 0806 20:35:09.342348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.345641 (+ 3293us) service_pool.cc:227] Handling call
0806 20:35:13.048686 (+3703045us) spinlock_profiling.cc:243] Waited 88.6 ms on lock 0x7b5400328054. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.048859 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":203676544}
I20260806 20:35:13.051942 14477 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:13.053467 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1441) took 3573 ms. Trace:
I20260806 20:35:13.054497 14477 rpcz_store.cc:276] 0806 20:35:09.480061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.051500 (+3571439us) service_pool.cc:227] Handling call
0806 20:35:13.051767 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.053240 (+ 1473us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.053420 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:13.071560 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.583s user 0.013s sys 0.023s
W20260806 20:35:13.072106 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.584s user 0.013s sys 0.023s
W20260806 20:35:13.072444 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 3.58403e+06us
I20260806 20:35:13.100742 14477 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:13.102738 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1442) took 3622 ms. Trace:
I20260806 20:35:13.103947 14477 rpcz_store.cc:276] 0806 20:35:09.480509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.055898 (+3575389us) service_pool.cc:227] Handling call
0806 20:35:13.100528 (+ 44630us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.102428 (+ 1900us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.102677 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.107496 14475 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:13.107853 14520 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.533s user 0.012s sys 0.033s
W20260806 20:35:13.108362 14520 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.548s user 0.012s sys 0.033s
I20260806 20:35:13.119029 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1440) took 3650 ms. Trace:
I20260806 20:35:13.120501 14474 rpcz_store.cc:276] 0806 20:35:09.468812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.050867 (+3582055us) service_pool.cc:227] Handling call
0806 20:35:13.116107 (+ 65240us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.116399 (+ 292us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.116442 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.116531 (+ 89us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.116601 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.116690 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.118735 (+ 2045us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b54002af650. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.118929 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":149090048}
W20260806 20:35:11.128961 16077 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.956s user 0.007s sys 0.019s
I20260806 20:35:13.123162 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1447) took 3640 ms. Trace:
W20260806 20:35:13.123694 14478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1332) took 3782 ms (client timeout 2212 ms). Trace:
W20260806 20:35:13.189251 14478 rpcz_store.cc:269] 0806 20:35:09.341640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.348108 (+ 6468us) service_pool.cc:227] Handling call
0806 20:35:13.115893 (+3767785us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54002af654. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.123621 (+ 7728us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":376813440}
W20260806 20:35:13.123993 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.695s user 0.022s sys 0.046s
I20260806 20:35:13.124312 15908 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.127233 16052 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:13.128101 14554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.038s user 0.024s sys 0.056s
W20260806 20:35:13.191401 14554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.101s user 0.025s sys 0.057s
W20260806 20:35:13.138043 14470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.833s user 0.012s sys 0.032s
W20260806 20:35:13.166903 14471 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 3.072s user 0.014s sys 0.047s
I20260806 20:35:13.167757 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.192337 14630 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:13.170660 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.171376 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1382) took 5408 ms. Trace:
I20260806 20:35:13.174034 14632 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17eec21790894c559ad9623765626416"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:35:13.193681 14632 raft_consensus.cc:595] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:13.193477 14629 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:13.194295 14632 raft_consensus.cc:603] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:13.175974 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1389) took 5083 ms. Trace:
I20260806 20:35:13.195441 14551 rpcz_store.cc:276] 0806 20:35:08.093043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:08.094086 (+ 1043us) service_pool.cc:227] Handling call
0806 20:35:08.094354 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:09.673944 (+1579590us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.175676 (+3501732us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540013a1d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:13.175903 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3436452608}
W20260806 20:35:13.177408 15153 connection.cc:232] RPC call timeout handler was delayed by -2.1656589100020938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18700000000000003s for a response
W20260806 20:35:13.197081 15153 connection.cc:232] RPC call timeout handler was delayed by -1.3022167049930431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2822s for a response
W20260806 20:35:13.197532 15153 connection.cc:232] RPC call timeout handler was delayed by -1.2687863009923603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20260806 20:35:11.119834 16000 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.006s sys 0.020s
I20260806 20:35:13.199694 16037 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:13.200206 14473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1328) took 5197 ms (client timeout 3287 ms). Trace:
I20260806 20:35:13.200345 16037 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:13.213148 14473 rpcz_store.cc:269] 0806 20:35:08.002712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:08.444144 (+441432us) service_pool.cc:227] Handling call
0806 20:35:13.199915 (+4755771us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400107cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.200116 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3755928960}
I20260806 20:35:13.215535 16037 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:13.193447 14553 rpcz_store.cc:276] 0806 20:35:07.763485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.764767 (+ 1282us) service_pool.cc:227] Handling call
0806 20:35:07.803151 (+ 38384us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:10.270705 (+2467554us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.171022 (+2900317us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54000a8f54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:13.171281 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5343407360}
I20260806 20:35:13.208779 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1494) took 3043 ms. Trace:
I20260806 20:35:13.219601 14628 rpcz_store.cc:276] 0806 20:35:10.165537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.207675 (+3042138us) service_pool.cc:227] Handling call
0806 20:35:13.208097 (+ 422us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.208321 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.208359 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.208413 (+ 54us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.208491 (+ 78us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.208564 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.208709 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.192349 14474 rpcz_store.cc:276] 0806 20:35:09.486150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.121971 (+3635821us) service_pool.cc:227] Handling call
0806 20:35:13.122375 (+ 404us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:13.122762 (+ 387us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.122803 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.122852 (+ 49us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.122892 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.122961 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.123104 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21170,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":87}
W20260806 20:35:13.224785 16043 consensus_meta.cc:287] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 2.878s user 0.021s sys 0.013s
I20260806 20:35:13.225468 16043 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:13.227438 14441 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.165s user 0.019s sys 0.025s
W20260806 20:35:13.238157 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.656s user 0.024s sys 0.027s
W20260806 20:35:13.239969 16077 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.067s user 0.013s sys 0.029s
W20260806 20:35:13.250124 14441 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.188s user 0.020s sys 0.025s
W20260806 20:35:13.256381 14549 consensus_meta.cc:287] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 2.974s user 0.017s sys 0.015s
I20260806 20:35:13.256879 14549 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:13.257485 14549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1221) took 3005 ms (client timeout 2218 ms). Trace:
W20260806 20:35:13.257858 14549 rpcz_store.cc:269] 0806 20:35:10.251605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:10.251714 (+ 109us) service_pool.cc:227] Handling call
0806 20:35:13.257421 (+3005707us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:13.258363 16043 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.260387 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1445) took 3778 ms. Trace:
I20260806 20:35:13.260818 14477 rpcz_store.cc:276] 0806 20:35:09.482004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:13.105443 (+3623439us) service_pool.cc:227] Handling call
0806 20:35:13.106067 (+ 624us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:13.106409 (+ 342us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:13.106438 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:13.106908 (+ 470us) log.cc:842] Serialized 26 byte log entry
0806 20:35:13.253177 (+146269us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.253215 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.253290 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:13.260220 (+ 6930us) raft_consensus.cc:1671] finished
0806 20:35:13.260238 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.260336 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":37241,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"thread_start_us":145698,"threads_started":1,"wal-append.queue_time_us":145382}
I20260806 20:35:13.260428 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54486 (request call id 1199) took 4514 ms. Trace:
I20260806 20:35:13.264552 14550 rpcz_store.cc:276] 0806 20:35:08.746425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:08.746636 (+ 211us) service_pool.cc:227] Handling call
0806 20:35:13.259734 (+4513098us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54000a8f54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b8438c6e 00007fb9b1b2f807 00007fb9b1b2f792 00007fb9b1b2f6c1 00007fb9b1b2f63c 00007fb9b1b2e8f2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.260352 (+ 618us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3171173248}
W20260806 20:35:13.279572 14633 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 3.097s user 0.011s sys 0.020s
W20260806 20:35:13.285280 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.845s user 0.025s sys 0.025s
I20260806 20:35:13.284098 16048 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:11.036095 15722 consensus_queue.cc:237] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:13.292096 14631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.085s user 0.003s sys 0.021s
W20260806 20:35:13.337204 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 323790us
W20260806 20:35:13.339226 14444 connection.cc:232] RPC call timeout handler was delayed by -2.5947668099834118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20400000000000001s for a response
W20260806 20:35:13.339423 14470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.034s user 0.013s sys 0.033s
W20260806 20:35:13.340010 14444 connection.cc:232] RPC call timeout handler was delayed by -2.5515668099978939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2088s for a response
I20260806 20:35:13.341342 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 115) took 4210 ms. Trace:
W20260806 20:35:13.342155 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.902s user 0.026s sys 0.025s
W20260806 20:35:13.342581 14471 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.247s user 0.015s sys 0.048s
W20260806 20:35:13.343874 14521 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.774s user 0.012s sys 0.034s
W20260806 20:35:13.348834 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.919s user 0.023s sys 0.046s
W20260806 20:35:13.350003 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1336) took 4006 ms (client timeout 1810 ms). Trace:
W20260806 20:35:13.350526 14475 rpcz_store.cc:269] 0806 20:35:09.343148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.346890 (+ 3742us) service_pool.cc:227] Handling call
0806 20:35:13.349301 (+4002411us) spinlock_profiling.cc:243] Waited 3.7 s on lock 0x7b54003f43d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.349688 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8500921600}
I20260806 20:35:13.352950 15946 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:13.354169 16030 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:13.353920 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.354379 15946 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.355533 14627 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:13.355352 16030 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:13.344297 14521 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.775s user 0.013s sys 0.034s
W20260806 20:35:13.357553 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.776s user 0.025s sys 0.028s
W20260806 20:35:13.358265 14444 connection.cc:232] RPC call timeout handler was delayed by -2.5140336400072556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260806 20:35:13.358305 14631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.151s user 0.003s sys 0.022s
W20260806 20:35:13.363654 14444 connection.cc:232] RPC call timeout handler was delayed by -2.4345668099995237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
W20260806 20:35:13.377420 14444 connection.cc:232] RPC call timeout handler was delayed by -2.3535668099939357s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2308s for a response
W20260806 20:35:13.390010 14444 connection.cc:232] RPC call timeout handler was delayed by -2.2629645490087569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260806 20:35:13.392153 14444 connection.cc:232] RPC call timeout handler was delayed by -2.1803219370194711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20260806 20:35:13.393479 14444 connection.cc:232] RPC call timeout handler was delayed by -2.0754668099980336s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2617s for a response
W20260806 20:35:13.395879 14444 connection.cc:232] RPC call timeout handler was delayed by -2.0509500950283837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20260806 20:35:13.397131 14444 connection.cc:232] RPC call timeout handler was delayed by -2.0358068390050903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
W20260806 20:35:13.397420 14444 connection.cc:232] RPC call timeout handler was delayed by -2.0349595090083312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20320000000000002s for a response
I20260806 20:35:13.398855 16043 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:13.398998 15983 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:13.399748 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1488) took 3240 ms. Trace:
W20260806 20:35:13.394493 16000 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.214s user 0.010s sys 0.037s
W20260806 20:35:13.400846 14633 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.218s user 0.013s sys 0.020s
I20260806 20:35:13.341821 14470 rpcz_store.cc:276] 0806 20:35:09.130543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.130706 (+ 163us) service_pool.cc:227] Handling call
0806 20:35:09.131759 (+ 1053us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:09.131974 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:10.305122 (+1173148us) log.cc:842] Serialized 14 byte log entry
0806 20:35:13.340342 (+3035220us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.340499 (+ 157us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:13.340701 (+ 202us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.340918 (+ 217us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.341204 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1614274,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":16979,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":360,"thread_start_us":4203122,"threads_started":2,"wal-append.queue_time_us":1497814,"wal-append.run_cpu_time_us":1310,"wal-append.run_wall_time_us":1001332}
I20260806 20:35:13.400118 15983 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.403566 16030 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:13.404318 14444 connection.cc:232] RPC call timeout handler was delayed by -2.0214668099943083s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2677s for a response
W20260806 20:35:13.404902 14444 connection.cc:232] RPC call timeout handler was delayed by -1.8534582350112032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
I20260806 20:35:13.405433 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.405875 16054 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.406383 14627 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:13.406589 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.407219 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:13.405686 14444 connection.cc:232] RPC call timeout handler was delayed by -1.7802360150089953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2238s for a response
W20260806 20:35:13.408032 14444 connection.cc:232] RPC call timeout handler was delayed by -1.7219343010219745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2864s for a response
W20260806 20:35:13.408420 14444 connection.cc:232] RPC call timeout handler was delayed by -1.6716582349909004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260806 20:35:13.400058 16043 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:13.408761 14444 connection.cc:232] RPC call timeout handler was delayed by -1.658347793010762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20260806 20:35:13.408675 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:13.409171 14444 connection.cc:232] RPC call timeout handler was delayed by -1.532208710006671s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260806 20:35:13.409554 14444 connection.cc:232] RPC call timeout handler was delayed by -1.4208769770048093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260806 20:35:13.409879 14444 connection.cc:232] RPC call timeout handler was delayed by -1.3797402930213138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21730000000000002s for a response
I20260806 20:35:13.406921 16054 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:13.410279 14444 connection.cc:232] RPC call timeout handler was delayed by -1.3762792060151696s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
W20260806 20:35:13.414093 14444 connection.cc:232] RPC call timeout handler was delayed by -1.041042552998988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
W20260806 20:35:13.414464 14444 connection.cc:232] RPC call timeout handler was delayed by -1.0191509530122858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
I20260806 20:35:13.415272 16110 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:13.416219 16110 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.419085 16110 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.424827 16029 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.425133 15908 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:13.425769 16029 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.426074 15908 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.428268 15908 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.428630 16057 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:13.429582 16057 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.433736 16054 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.435644 14625 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:13.439145 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 109) took 5298 ms. Trace:
W20260806 20:35:13.439689 14442 connection.cc:232] RPC call timeout handler was delayed by -2.0393827880034223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260806 20:35:13.440158 14442 connection.cc:232] RPC call timeout handler was delayed by -1.8498130140069406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
I20260806 20:35:13.440243 16043 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:13.440515 14442 connection.cc:232] RPC call timeout handler was delayed by -1.3723486410162877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
W20260806 20:35:13.440928 14442 connection.cc:232] RPC call timeout handler was delayed by -1.2616486410261132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
I20260806 20:35:13.441753 15722 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.441910 14442 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:13.442747 15722 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.443446 14442 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:13.446305 14479 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.428212 16029 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:13.450150 14441 connection.cc:232] RPC call timeout handler was delayed by -2.0681371680111624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
I20260806 20:35:13.400267 14631 rpcz_store.cc:276] 0806 20:35:10.159742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:10.166169 (+ 6427us) service_pool.cc:227] Handling call
0806 20:35:10.205143 (+ 38974us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:10.205877 (+ 734us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:10.205956 (+ 79us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:10.206847 (+ 891us) log.cc:842] Serialized 26 byte log entry
0806 20:35:13.399011 (+3192164us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:13.399122 (+ 111us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.399266 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:13.399346 (+ 80us) raft_consensus.cc:1671] finished
0806 20:35:13.399365 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.399625 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":26478,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":43940,"thread_start_us":3165769,"threads_started":1,"wal-append.queue_time_us":1603057,"wal-append.run_cpu_time_us":1783,"wal-append.run_wall_time_us":1001810}
I20260806 20:35:13.439716 14476 rpcz_store.cc:276] 0806 20:35:08.140511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:08.644185 (+503674us) service_pool.cc:227] Handling call
0806 20:35:08.658957 (+ 14772us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:08.659588 (+ 631us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:08.659642 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:08.688995 (+ 29353us) log.cc:842] Serialized 26 byte log entry
0806 20:35:10.284863 (+1595868us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:10.284899 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:10.284991 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:13.438507 (+3153516us) raft_consensus.cc:1671] finished
0806 20:35:13.438560 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.438865 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93,"raft.queue_time_us":236,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":158,"thread_start_us":1576242,"threads_started":1,"wal-append.queue_time_us":1981244}
W20260806 20:35:13.486012 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1344) took 4068 ms (client timeout 2197 ms). Trace:
I20260806 20:35:13.488698 15908 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:13.492762 15996 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 3.736s user 0.017s sys 0.024s
I20260806 20:35:13.490223 16029 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:13.492873 16043 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:13.492362 14479 rpcz_store.cc:269] 0806 20:35:09.417614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.484306 (+ 66692us) service_pool.cc:227] Handling call
0806 20:35:13.448412 (+3964106us) spinlock_profiling.cc:243] Waited 3.92 s on lock 0x7b5400116cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.485950 (+ 37538us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9018728832}
I20260806 20:35:13.494175 15722 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.493644 16029 raft_consensus.cc:686] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:13.491647 15151 connection.cc:232] RPC call timeout handler was delayed by -1.6130369269812945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1925s for a response
W20260806 20:35:13.496167 15151 connection.cc:232] RPC call timeout handler was delayed by -1.5669072070159018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2822s for a response
W20260806 20:35:13.496714 15151 connection.cc:232] RPC call timeout handler was delayed by -1.5574302989989519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
I20260806 20:35:13.483383 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1429) took 5681 ms. Trace:
W20260806 20:35:13.493203 15996 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.822s user 0.018s sys 0.025s
I20260806 20:35:13.493008 15908 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:13.498852 15908 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.498272 14471 rpcz_store.cc:276] 0806 20:35:07.802435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.802632 (+ 197us) service_pool.cc:227] Handling call
0806 20:35:07.827895 (+ 25263us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:08.571592 (+743697us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:08.571625 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:08.586997 (+ 15372us) log.cc:842] Serialized 26 byte log entry
0806 20:35:10.075376 (+1488379us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:13.343374 (+3267998us) log.cc:842] Serialized 14 byte log entry
0806 20:35:13.410707 (+ 67333us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.421081 (+ 10374us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:13.421155 (+ 74us) raft_consensus.cc:1671] finished
0806 20:35:13.421172 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.483277 (+ 62105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11344,"prepare.queue_time_us":798357,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":83,"raft.queue_time_us":84918,"raft.run_cpu_time_us":552,"raft.run_wall_time_us":16534,"thread_start_us":5468462,"threads_started":4,"wal-append.queue_time_us":1909502,"wal-append.run_cpu_time_us":28481,"wal-append.run_wall_time_us":2467043}
I20260806 20:35:13.503665 15961 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:13.490209 14442 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:13.508606 16051 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.504678 15961 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.525209 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.526427 14631 raft_consensus.cc:2396] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:13.528247 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.529631 16029 consensus_queue.cc:237] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.531195 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.532195 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1486) took 3394 ms. Trace:
I20260806 20:35:13.532347 14631 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:13.536989 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.538344 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.541097 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.542662 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.548961 14626 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:13.551666 14625 raft_consensus.cc:2396] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:13.552076 14628 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:13.552006 15722 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:13.553372 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 70508.6us
I20260806 20:35:13.553287 15722 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.553602 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.554737 14479 raft_consensus.cc:2354] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:13.555871 16119 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:13.555974 15722 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:13.555891 16114 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.152s user 0.002s sys 0.000s
I20260806 20:35:13.512883 15983 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.539686 15821 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.559367 15821 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.561859 16119 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.564002 15821 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.564167 16119 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.560796 14441 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:13.556825 16051 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:13.567296 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.568296 15996 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:13.568571 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.569141 15996 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.569598 14475 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:13.571611 15996 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:13.574479 14595 connection.cc:232] RPC call timeout handler was delayed by -2.993831048021093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20260806 20:35:13.574815 14595 connection.cc:232] RPC call timeout handler was delayed by -2.9189889080007561s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
W20260806 20:35:13.575517 14595 connection.cc:232] RPC call timeout handler was delayed by -2.8075974909879733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
I20260806 20:35:13.548106 14633 rpcz_store.cc:276] 0806 20:35:10.137551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:10.137718 (+ 167us) service_pool.cc:227] Handling call
0806 20:35:10.168672 (+ 30954us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:10.168926 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:13.401520 (+3232594us) log.cc:842] Serialized 14 byte log entry
0806 20:35:13.467314 (+ 65794us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.482382 (+ 15068us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:13.500487 (+ 18105us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.502200 (+ 1713us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.532120 (+ 29920us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1614117,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":27516,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":274,"thread_start_us":3281905,"threads_started":2,"wal-append.queue_time_us":98236}
W20260806 20:35:13.575861 14595 connection.cc:232] RPC call timeout handler was delayed by -2.7364234299748205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
I20260806 20:35:13.575295 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.577219 14627 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:35:13.577821 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 67359.6us
W20260806 20:35:13.576205 14595 connection.cc:232] RPC call timeout handler was delayed by -2.6925967579882126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1978s for a response
W20260806 20:35:13.578783 14595 connection.cc:232] RPC call timeout handler was delayed by -2.5944259260140825s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20260806 20:35:13.578992 14595 connection.cc:232] RPC call timeout handler was delayed by -2.4669467909843661s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
W20260806 20:35:13.579149 14595 connection.cc:232] RPC call timeout handler was delayed by -2.3893247350060847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3287s for a response
W20260806 20:35:13.579291 14595 connection.cc:232] RPC call timeout handler was delayed by -2.2304942040063906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19940000000000002s for a response
W20260806 20:35:13.579435 14595 connection.cc:232] RPC call timeout handler was delayed by -2.2161610060138628s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.219s for a response
W20260806 20:35:13.579574 14595 connection.cc:232] RPC call timeout handler was delayed by -2.2097569999750704s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
W20260806 20:35:13.579725 14595 connection.cc:232] RPC call timeout handler was delayed by -2.180998655006988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
W20260806 20:35:13.579869 14595 connection.cc:232] RPC call timeout handler was delayed by -2.1489955589931924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
W20260806 20:35:13.580008 14595 connection.cc:232] RPC call timeout handler was delayed by -2.0420758700056467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21330000000000002s for a response
W20260806 20:35:13.580150 14595 connection.cc:232] RPC call timeout handler was delayed by -1.8280720790207852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32820000000000005s for a response
I20260806 20:35:13.589876 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.580933 15821 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:13.590807 15821 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:13.591310 15821 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:13.567482 16043 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.588837 15151 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:13.586555 16051 raft_consensus.cc:686] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:13.607419 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.609798 14595 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:13.578531 15908 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:13.615646 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 112) took 4616 ms (client timeout 2068 ms). Trace:
I20260806 20:35:13.615660 15983 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.622953 16051 consensus_queue.cc:237] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.623680 15983 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:13.623348 14469 rpcz_store.cc:269] 0806 20:35:08.998967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.002225 (+ 3258us) service_pool.cc:227] Handling call
0806 20:35:13.615188 (+4612963us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54001068d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:13.615413 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":256013568}
I20260806 20:35:13.626667 15983 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:13.616170 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.627506 14475 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:35:13.629407 16043 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.630533 16057 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.630618 15908 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:13.631126 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 69849.4us
I20260806 20:35:13.631575 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.633617 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.634456 15180 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
W20260806 20:35:13.635967 14404 debug-util.cc:398] Leaking SignalData structure 0x7b080019b6e0 after lost signal to thread 14407
W20260806 20:35:13.636543 14404 debug-util.cc:398] Leaking SignalData structure 0x7b080019bb60 after lost signal to thread 14421
I20260806 20:35:13.636693 15185 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:13.636883 16043 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:13.637470 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.638200 15183 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:13.638358 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.639248 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.639664 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.640240 14478 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:13.640103 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.640509 14553 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.641376 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.641885 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.642086 15180 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:13.642467 14478 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:13.645171 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.646673 16028 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:13.647182 16028 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.648666 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.649653 15179 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:13.617786 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.618229 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:13.618469 14596 connection.cc:232] RPC call timeout handler was delayed by -2.6215332019783091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1978s for a response
I20260806 20:35:13.620128 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.645146 15983 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:13.657609 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.660645 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.659435 15983 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:13.662048 15983 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:13.663195 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.616529 16122 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.644675 14595 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:13.657414 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.664379 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:13.659117 14596 connection.cc:232] RPC call timeout handler was delayed by -2.047679661976872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
I20260806 20:35:13.607223 14370 catalog_manager.cc:5495] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:13.668282 16043 raft_consensus.cc:686] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:13.668387 14478 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:13.669817 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.670616 14478 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:13.649609 14469 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:13.677069 16130 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:13.677469 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:13.604063 16119 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:13.683018 14479 raft_consensus.cc:2379] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.681238 16122 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.683498 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.685753 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.688755 14471 raft_consensus.cc:2354] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:13.690521 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.692577 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.690568 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.669869 16043 consensus_queue.cc:237] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.695209 16058 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.695889 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 160) took 4515 ms. Trace:
I20260806 20:35:13.705408 14478 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:13.696034 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.681393 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.706768 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.707396 14478 raft_consensus.cc:2354] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:13.707865 14477 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:13.707801 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.710394 14471 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.712054 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.712743 14474 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:13.713470 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.713689 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.714829 15821 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.716897 14473 raft_consensus.cc:2379] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.714908 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.712440 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.718927 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.714859 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.721213 14553 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:13.726910 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.728514 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.729400 14549 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:13.736411 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.735555 16130 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:13.730311 14548 raft_consensus.cc:2354] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:13.730722 16058 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.728070 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.725443 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:13.740932 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.747064 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.741379 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.750885 15150 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:13.705384 14554 rpcz_store.cc:276] 0806 20:35:09.180775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:09.186171 (+ 5396us) service_pool.cc:227] Handling call
0806 20:35:09.186646 (+ 475us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:09.186872 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:10.090148 (+903276us) log.cc:842] Serialized 14 byte log entry
0806 20:35:13.695250 (+3605102us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:13.695340 (+ 90us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:13.695424 (+ 84us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:13.695516 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:13.695703 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":503150,"prepare.queue_time_us":1157448,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":50,"raft.queue_time_us":504485,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":241,"thread_start_us":3991037,"threads_started":2,"wal-append.queue_time_us":1304588,"wal-append.run_cpu_time_us":4179,"wal-append.run_wall_time_us":2097307}
I20260806 20:35:13.754583 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.754369 14370 catalog_manager.cc:5495] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:13.742054 14552 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:13.756389 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.742657 16058 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.742707 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.743350 14551 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.762765 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:13.763495 15183 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.745018 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.743489 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.757118 14555 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:13.759867 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.760109 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.760978 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.773240 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.719836 16119 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.740149 15996 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:13.755945 14469 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:13.781059 14546 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.782639 14479 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.783260 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.750905 15946 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.800717 15150 leader_election.cc:398] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:13.804860 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.807952 15150 leader_election.cc:401] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 20:35:13.799172 15184 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:13.783735 16122 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.785224 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.799229 14551 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:13.818404 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 141140us
I20260806 20:35:13.819365 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:13.820552 14596 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:13.821708 15185 raft_consensus.cc:2354] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
I20260806 20:35:13.822861 16119 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:13.823897 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 716744us
I20260806 20:35:13.824741 15182 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
W20260806 20:35:13.824640 14521 connection.cc:232] RPC call timeout handler was delayed by -2.7790587849740405s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20260806 20:35:13.784741 16127 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:13.784667 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:13.825192 14521 connection.cc:232] RPC call timeout handler was delayed by -2.3974587849806994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19710000000000003s for a response
W20260806 20:35:13.828521 14521 connection.cc:232] RPC call timeout handler was delayed by -1.8449844989809208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2456s for a response
W20260806 20:35:13.828804 14521 connection.cc:232] RPC call timeout handler was delayed by -1.2877475710120052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29900000000000004s for a response
I20260806 20:35:13.829744 16067 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.829829 16077 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.820572 15983 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.825951 14518 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:13.834064 16024 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:13.830667 16077 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.834791 16024 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.836642 16077 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.837678 16077 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:13.827147 15821 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.816655 14475 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:13.842691 15918 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:13.842824 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.843358 15918 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.844158 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.840430 16127 raft_consensus.cc:686] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:13.844913 14552 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:13.845504 15918 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.845865 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.846185 15918 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:13.839138 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 52653.1us
I20260806 20:35:13.847113 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.840425 15996 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:13.845566 16127 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.842060 16093 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:13.826740 14546 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:13.851193 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.855360 15983 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:13.844211 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:13.825057 16028 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.857954 15996 consensus_queue.cc:1035] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:13.846823 15918 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.858800 16119 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:35:13.830400 16067 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:13.876258 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 50449.8us
I20260806 20:35:13.859896 16024 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.860656 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.862553 15961 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:13.863139 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.863336 15837 raft_consensus.cc:3032] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:13.882661 14633 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:13.864022 14548 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 20:35:13.872679 16093 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.877816 15957 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:13.879040 16028 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.879710 14474 raft_consensus.cc:2354] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:13.885027 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:13.885569 14404 debug-util.cc:398] Leaking SignalData structure 0x7b0800349800 after lost signal to thread 14503
I20260806 20:35:13.885684 16067 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:13.866343 16077 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.883796 14550 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:13.889029 15957 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:13.890856 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2160 after lost signal to thread 14504
I20260806 20:35:13.891042 15957 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:13.892370 14404 debug-util.cc:398] Leaking SignalData structure 0x7b080033b7c0 after lost signal to thread 14581
I20260806 20:35:13.895761 15918 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:13.896248 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0760 after lost signal to thread 14657
I20260806 20:35:13.891120 15860 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:13.901051 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0740 after lost signal to thread 14658
I20260806 20:35:13.901166 14441 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:13.902637 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08002911a0 after lost signal to thread 15212
W20260806 20:35:13.902938 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1780 after lost signal to thread 15213
I20260806 20:35:13.903031 14441 leader_election.cc:398] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
I20260806 20:35:13.903404 14441 leader_election.cc:401] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:13.887112 15837 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:13.904628 15837 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.905488 16031 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:13.894032 16093 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.906183 16031 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
I20260806 20:35:13.907729 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.908324 14627 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:13.914276 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 95072.5us
I20260806 20:35:13.920720 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.920959 14441 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:13.922420 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.920929 15837 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.923157 14626 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:13.927000 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:13.927750 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.928231 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.928875 14624 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:13.899248 15150 leader_election.cc:398] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:13.929768 15150 leader_election.cc:401] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 20:35:13.929560 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:13.930014 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.930495 15179 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:13.931176 14552 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:13.933262 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.937196 14555 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:13.936715 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 47010.5us
I20260806 20:35:13.939837 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.940829 14628 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:13.941689 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.933203 16110 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.942741 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.943356 14546 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:13.941639 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.929131 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:13.946163 14555 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:13.946841 15183 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:13.947465 16129 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:13.947431 14595 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:13.948755 16119 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:13.949484 14441 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:13.952875 16132 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:13.956888 16132 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:13.957156 16110 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:13.955319 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:13.958729 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.957384 16132 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.959893 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.960511 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.960790 14549 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:35:13.962520 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.959002 14595 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:13.963387 14548 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:13.963411 16119 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:13.963912 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.964658 16031 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:13.965243 16031 raft_consensus.cc:686] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:13.965991 16077 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:13.966667 14550 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.966369 16031 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:13.961618 15957 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.965727 14595 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:13.962618 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:13.969076 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.969461 14555 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.970158 16119 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:13.970276 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.970404 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.971544 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:13.972095 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.972373 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:13.973249 14551 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:13.973057 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:13.973093 14626 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:13.972206 14627 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:13.973901 14628 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.971710 16119 consensus_queue.cc:1046] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260806 20:35:13.977316 15722 raft_consensus.cc:3032] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:13.977706 15722 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:13.978502 14595 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:13.964313 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.978631 15722 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:13.982226 16119 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:13.982726 16119 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:13.956776 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.983265 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.983666 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.984218 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.984797 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.985345 15722 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:13.986244 14547 raft_consensus.cc:1261] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:13.985350 16119 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:13.987362 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.996210 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:13.982694 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:14.000900 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 7f41753baced49a1997940dfc28b2e00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:14.002426 15178 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.005147 14546 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:14.005548 14441 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:14.011950 15946 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:14.007982 14595 leader_election.cc:398] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3.
I20260806 20:35:14.012502 14595 leader_election.cc:401] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260806 20:35:14.008780 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 75262.6us
I20260806 20:35:14.009091 14596 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.010720 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:14.008769 16132 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.006328 14555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:14.015690 14555 raft_consensus.cc:595] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:35:14.016069 14555 raft_consensus.cc:603] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:35:13.981452 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08003740e0 after lost signal to thread 15981
I20260806 20:35:14.012598 15946 raft_consensus.cc:686] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.017043 14442 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:14.009932 15957 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:14.018671 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.017547 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.019711 16028 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:14.021000 15953 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.025362 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:14.026875 14624 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.026772 16136 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 20:35:14.027267 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.028482 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:14.029346 16132 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:14.026913 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:14.029409 15961 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.030656 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.031291 14549 raft_consensus.cc:2362] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4. Re-sending same reply.
I20260806 20:35:14.034096 14474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:35:14.035769 14628 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.037055 15946 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.024626 16030 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:14.040709 16077 consensus_queue.cc:1035] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:14.059748 16136 consensus_queue.cc:1046] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:14.023562 15821 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:14.059195 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.045035 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.060959 14546 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.046856 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.061286 14548 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.062258 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:14.063519 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08003c21e0 after lost signal to thread 16036
I20260806 20:35:14.047780 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:14.063880 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08003c21a0 after lost signal to thread 16038
W20260806 20:35:14.064167 14404 debug-util.cc:398] Leaking SignalData structure 0x7b08003c2180 after lost signal to thread 16039
I20260806 20:35:14.047142 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.048676 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.064729 14633 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.064620 14442 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:14.052784 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 45017us
I20260806 20:35:14.053434 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.056731 14596 leader_election.cc:398] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Vote denied by peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) with higher term. Message: Invalid argument: T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:14.057588 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.059059 16067 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:14.060060 16030 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:14.046130 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.060736 15821 raft_consensus.cc:686] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.085665 15821 consensus_queue.cc:237] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.063417 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.047601 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.087826 14625 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:14.065516 14630 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.048741 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.064369 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:14.088059 14548 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.064618 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.094084 14632 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.094854 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.066823 14596 leader_election.cc:401] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 20:35:14.051182 16119 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.097311 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:14.098126 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 56082.6us
I20260806 20:35:14.098570 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.099264 14595 leader_election.cc:398] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:14.099879 14632 raft_consensus.cc:2379] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 2: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.072961 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.102242 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.102906 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:14.103147 14469 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:14.103173 15181 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:14.105170 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.106295 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.100639 14633 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.106295 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.079190 16077 raft_consensus.cc:3032] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:14.080648 14624 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.114832 15996 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:14.113762 14474 raft_consensus.cc:595] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:35:14.115741 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.117024 14624 raft_consensus.cc:2396] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:14.118201 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.072847 16030 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.120012 14470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc270596f078400e9e1b181a9706b516"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "148bf2a11dba423ab364371f0458e871"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:35:14.120410 14595 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:14.120745 14470 raft_consensus.cc:595] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Received request to transfer leadership to TS 148bf2a11dba423ab364371f0458e871
I20260806 20:35:14.121146 14470 raft_consensus.cc:603] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:14.121663 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.122712 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.123179 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:35:14.123344 14633 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.124156 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.081458 16051 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:14.124949 14633 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.125234 16051 raft_consensus.cc:686] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.125883 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:35:14.125775 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.126874 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.052117 15179 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:14.088404 14629 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.090975 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.092715 14550 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.097483 16119 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:14.073033 14552 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260806 20:35:14.133190 14552 pending_rounds.cc:76] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 4
I20260806 20:35:14.103749 14475 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:14.073668 14442 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:14.064643 14404 debug-util.cc:398] Leaking SignalData structure 0x7b0800381ac0 after lost signal to thread 16066
I20260806 20:35:14.126600 16051 consensus_queue.cc:237] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.073711 14628 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.127533 16130 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:14.113102 16077 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:14.119676 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.128068 15996 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:14.128582 14546 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.120083 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.130838 14633 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.129469 14631 raft_consensus.cc:2396] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:14.145684 16077 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.146858 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.147135 15953 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3.
I20260806 20:35:14.147526 14631 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.137527 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.149535 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.150243 14628 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.151091 14624 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.133435 14552 pending_rounds.cc:97] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260806 20:35:14.140913 14441 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:14.152630 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.144771 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.143941 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.155326 14633 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.155961 16029 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:14.156500 16029 raft_consensus.cc:686] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.143651 16132 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:14.157429 16029 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:14.144609 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 71037.7us
I20260806 20:35:14.153839 16122 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:14.142480 16130 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:14.164693 16130 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:14.164844 14546 raft_consensus.cc:1261] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:14.140699 14404 debug-util.cc:398] Leaking SignalData structure 0x7b0800361d20 after lost signal to thread 16068
W20260806 20:35:14.168891 14404 debug-util.cc:398] Leaking SignalData structure 0x7b0800340ca0 after lost signal to thread 16075
I20260806 20:35:14.147081 16077 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:14.171052 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.171912 14546 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:14.142019 14595 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.172674 16031 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:14.173094 14550 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260806 20:35:14.171252 14548 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:35:14.152634 14552 raft_consensus.cc:2866] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:35:14.217676 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 58318.5us
I20260806 20:35:14.220438 14627 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.218255 14629 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.219870 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.222052 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.224762 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.225376 14470 raft_consensus.cc:2379] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:14.219102 14625 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:14.226287 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:14.234913 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 93076.2us
I20260806 20:35:14.199801 16031 raft_consensus.cc:686] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.238322 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.239009 14627 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.224843 16122 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.226842 15961 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) with higher term. Message: Invalid argument: T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:14.248101 14442 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:14.251976 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:14.222497 15996 consensus_queue.cc:1046] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 20:35:14.255455 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.258001 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.229460 15722 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260806 20:35:14.260792 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.261545 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.258275 14633 raft_consensus.cc:2362] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3. Re-sending same reply.
I20260806 20:35:14.262908 14442 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:14.240895 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.265009 14471 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:14.267774 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.272413 14633 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.267119 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:14.274371 15180 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:14.274703 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:14.275372 15186 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:14.276193 15996 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:14.276620 15996 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:14.276678 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:14.277098 15996 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.277945 16132 raft_consensus.cc:3032] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:14.278326 16132 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:14.278273 16030 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.284652 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.284839 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.285256 14633 raft_consensus.cc:2362] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4. Re-sending same reply.
I20260806 20:35:14.285746 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.286314 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.286964 15946 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:14.287279 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:14.279404 16132 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.280576 15722 raft_consensus.cc:3032] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:14.280761 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.283073 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.283390 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.305382 14630 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.284082 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.305425 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.284370 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.284756 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.305961 14442 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:14.306293 14470 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:14.288893 14595 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.266800 14521 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.306845 14442 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:14.299754 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:14.255529 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:14.266885 16051 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:14.305894 14626 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.313210 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:14.313608 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 53844.1us
W20260806 20:35:14.314320 14518 connection.cc:232] RPC call timeout handler was delayed by -3.2538835860032123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
W20260806 20:35:14.314659 14518 connection.cc:232] RPC call timeout handler was delayed by -2.8722835860098712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19710000000000003s for a response
W20260806 20:35:14.314884 14518 connection.cc:232] RPC call timeout handler was delayed by -1.9380835860210937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3009s for a response
I20260806 20:35:14.314096 14628 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:14.282310 16031 consensus_queue.cc:237] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.286285 16030 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:14.312619 16151 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:14.304720 15722 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:14.334292 14595 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.334728 16067 raft_consensus.cc:2741] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:35:14.354601 14625 raft_consensus.cc:1261] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:14.316365 16122 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:14.342288 14624 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.319813 14477 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:14.374914 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 68127.9us
I20260806 20:35:14.376951 14628 raft_consensus.cc:1261] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:14.409519 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 44148.6us
I20260806 20:35:14.340355 16051 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:14.319674 16081 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:14.396612 16145 log.cc:924] Time spent T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
W20260806 20:35:14.364341 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 104032us
I20260806 20:35:14.335520 15946 raft_consensus.cc:686] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.360896 16151 raft_consensus.cc:686] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.423215 16051 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:14.375631 16132 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:14.428738 15953 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:14.430466 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:14.429699 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 95414.6us
I20260806 20:35:14.429512 15722 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.431520 14548 raft_consensus.cc:1261] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:14.432577 16148 log.cc:924] Time spent T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
W20260806 20:35:14.432942 16146 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20260806 20:35:14.433946 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:14.438772 15996 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.440385 16030 consensus_queue.cc:1035] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:14.441123 14476 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:14.447198 14518 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:14.397149 16143 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260806 20:35:14.463757 14550 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.450356 16081 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.461870 16051 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.481000 14521 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:14.396816 15837 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260806 20:35:14.488232 14630 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:14.480203 14595 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:14.493970 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47769.7us
I20260806 20:35:14.501140 16125 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:14.505573 16136 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:14.506117 16136 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:14.506575 16136 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.490168 15918 raft_consensus.cc:2741] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260806 20:35:14.558176 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 59968.7us
W20260806 20:35:14.577526 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 97499.1us
I20260806 20:35:14.477473 16151 consensus_queue.cc:237] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.609705 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:14.616130 14595 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.620010 14476 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 20:35:14.622116 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 178377us
I20260806 20:35:14.627671 16043 consensus_queue.cc:1046] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:14.630648 14477 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:14.631551 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.589040 16157 consensus_queue.cc:1035] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260806 20:35:14.632349 14626 raft_consensus.cc:2354] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:14.634658 14595 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.635749 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.637851 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.627805 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.585409 16130 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.646073 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.622068 15722 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.647655 15961 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.650005 14551 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:14.650802 16130 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:14.651265 14552 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:14.648603 14518 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.653292 14550 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:14.638762 15996 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:14.655611 14442 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:14.656682 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.659541 14595 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.657997 16157 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 20:35:14.660915 14595 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.665777 14626 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:35:14.665876 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.666863 14630 raft_consensus.cc:2396] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:14.667642 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.664611 15983 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.668663 16166 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 20:35:14.651753 15953 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.676364 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.658641 14624 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.681342 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.681965 14630 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:14.666143 16029 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:14.683298 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.685127 16164 raft_consensus.cc:3032] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:14.685350 14546 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:14.686297 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:14.687243 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:14.719224 14547 raft_consensus.cc:2362] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4. Re-sending same reply.
I20260806 20:35:14.689147 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:14.671288 15996 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 20:35:14.727851 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 65591.1us
I20260806 20:35:14.678457 16077 raft_consensus.cc:2741] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:14.733053 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.736992 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:14.736999 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 94720.1us
I20260806 20:35:14.737268 15151 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.737692 14630 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:14.735535 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 113846us
I20260806 20:35:14.649176 15946 consensus_queue.cc:237] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.706282 16164 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:14.730968 16029 raft_consensus.cc:686] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:14.735081 16081 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:14.756737 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.757445 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.758713 15151 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
W20260806 20:35:14.759866 16144 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.529s user 0.000s sys 0.005s
I20260806 20:35:14.760004 16029 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.761861 16122 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.775609 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.776042 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.778347 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.780984 14626 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:14.765573 15961 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:14.784883 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.784192 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.797122 14627 raft_consensus.cc:2354] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3.
I20260806 20:35:14.786123 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.798228 14625 raft_consensus.cc:2396] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260806 20:35:14.799836 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.777835 16110 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:14.809305 14625 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:35:14.816434 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.795285 15961 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:14.823583 15961 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:14.825726 16136 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:14.827525 14624 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:14.831466 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.834260 16136 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:14.846475 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:14.853632 14628 raft_consensus.cc:1226] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:14.856431 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 97776.2us
I20260806 20:35:14.857507 15151 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:14.858355 15151 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.860215 16130 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:14.860798 16130 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:14.861289 16130 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:14.863597 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 119365us
I20260806 20:35:14.864001 16182 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:14.864578 16182 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:14.865159 16182 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:14.860821 14595 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:14.868539 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 222926us
I20260806 20:35:14.869952 16125 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:14.870121 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.871376 14595 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:14.872956 16178 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.873085 16125 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:14.873708 16125 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:14.874197 16125 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:14.876616 14518 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:14.884979 14595 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:14.845392 16110 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:14.887521 14518 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:14.795931 16168 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260806 20:35:14.783255 16164 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.912302 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:14.888010 16077 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:14.912225 14595 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:14.829720 16051 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.877403 16178 consensus_queue.cc:1046] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 20:35:14.913779 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:14.910537 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:14.994387 16077 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:14.981416 16168 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:14.995021 16077 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:14.966189 14470 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:14.976583 16130 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.999111 16130 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.001461 14546 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:14.896950 16122 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:14.953799 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:14.976688 14550 raft_consensus.cc:3032] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:15.003233 14550 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:15.003414 15821 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:15.004062 14550 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.005211 14550 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:14.967512 16169 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260806 20:35:14.975980 15961 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.916525 14629 raft_consensus.cc:2362] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3. Re-sending same reply.
I20260806 20:35:15.018682 14469 raft_consensus.cc:1226] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:15.019730 16125 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.021682 16051 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.022081 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:15.022704 14549 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:15.022994 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 112509us
I20260806 20:35:14.977396 16136 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.025674 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:15.027592 16024 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.019778 16182 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:14.896790 16110 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:15.047421 16164 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:15.048895 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 136752us
I20260806 20:35:15.049067 14633 raft_consensus.cc:1226] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:15.050259 16110 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:15.051275 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:15.051885 14625 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:15.047485 16122 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.058842 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.059409 14625 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:15.060878 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.062060 14550 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 20:35:15.072192 14548 raft_consensus.cc:1261] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:15.074010 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 500956bd1a3846a3a7b16c51ddecbc44 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:15.077046 14521 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.079814 14521 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:15.081477 14521 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.120076 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.022706 16125 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:15.121299 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47205.5us
I20260806 20:35:15.123680 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.124511 14479 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:15.125284 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.122081 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.125975 14470 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:15.126468 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.127542 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.128693 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.128840 15961 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.077972 16199 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:15.130569 16172 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:15.131023 16172 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:15.131366 16172 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:15.132567 16156 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.003s sys 0.014s
W20260806 20:35:15.133106 16156 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.637s user 0.003s sys 0.014s
W20260806 20:35:15.133562 16190 log.cc:924] Time spent T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260806 20:35:15.130117 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.136972 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.055840 16178 consensus_queue.cc:1225] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer 148bf2a11dba423ab364371f0458e871 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 20:35:15.139292 14624 raft_consensus.cc:1261] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:15.136961 14549 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:15.143733 16024 raft_consensus.cc:3032] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:15.144192 16024 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:15.145407 15918 consensus_queue.cc:1046] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:15.149331 14478 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:15.150669 14442 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:15.137598 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.155488 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.156591 14518 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:15.157882 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 299571us
I20260806 20:35:15.158460 14595 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.145197 16024 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.161754 14595 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.121881 16067 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.164742 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.165773 16024 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:15.168977 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:15.169377 14478 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:15.165594 14471 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260806 20:35:15.160002 16178 raft_consensus.cc:3032] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:15.161552 16125 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.186285 14518 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.184515 14470 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:15.190537 15961 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:15.192616 16067 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.149212 15837 consensus_queue.cc:1046] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20260806 20:35:15.192978 14475 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:15.164373 16093 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.158582 16051 consensus_queue.cc:1035] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260806 20:35:15.195706 14624 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260806 20:35:15.193819 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 50197us
I20260806 20:35:15.192262 14518 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.190243 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:15.222191 14444 leader_election.cc:334] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.549s (SENT)
I20260806 20:35:15.222518 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.178512 16151 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 20:35:15.273795 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 68950.4us
W20260806 20:35:15.274101 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 68802.4us
W20260806 20:35:15.273794 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 83603.9us
I20260806 20:35:15.275334 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.281473 14595 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.282953 14595 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.189064 16178 raft_consensus.cc:729] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:15.291355 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 93619.1us
I20260806 20:35:15.196290 16175 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.101s
I20260806 20:35:15.295794 15151 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:15.296984 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:15.311833 16194 log.cc:924] Time spent T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260806 20:35:15.312767 14478 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
W20260806 20:35:15.302785 14444 leader_election.cc:334] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.596s (SENT)
I20260806 20:35:15.314102 14595 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:15.283057 16122 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:15.290584 16125 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.292635 16132 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.314338 14444 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.357374 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.354815 15953 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:15.362705 14549 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:15.302416 16151 raft_consensus.cc:686] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:15.365999 15953 raft_consensus.cc:686] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:15.366115 16193 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:15.317891 16182 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:15.369150 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 176959us
W20260806 20:35:15.374136 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 52423.9us
W20260806 20:35:15.401439 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 126912us
I20260806 20:35:15.402842 14518 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.409145 15151 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.365093 16093 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.415804 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.417460 14549 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:35:15.402253 16130 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:15.416523 14633 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:15.405167 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.411047 16122 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:15.412624 16178 consensus_queue.cc:260] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.428784 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:15.420413 16175 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.429585 16193 raft_consensus.cc:686] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:15.435933 16178 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:15.423134 16173 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.437230 16191 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:15.443331 15151 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:15.457381 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 165208us
I20260806 20:35:15.418392 16043 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20260806 20:35:15.464787 14444 leader_election.cc:334] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.772s (SENT)
I20260806 20:35:15.465482 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.421890 16130 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.456873 16081 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:15.421768 16110 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.425768 15953 consensus_queue.cc:237] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.428148 16122 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.427974 16077 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.430500 16029 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260806 20:35:15.471235 16182 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.471603 16191 raft_consensus.cc:686] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:15.476841 16030 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:15.493706 15153 leader_election.cc:334] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.870s (SENT)
I20260806 20:35:15.281250 16199 consensus_queue.cc:1046] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.204s
I20260806 20:35:15.438473 16179 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260806 20:35:15.498853 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.480206 16081 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:15.499159 14444 leader_election.cc:334] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.621s (SENT)
I20260806 20:35:15.439332 16180 consensus_queue.cc:1046] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260806 20:35:15.501044 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.504096 14444 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.499835 14550 raft_consensus.cc:1226] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:15.504444 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 103504us
I20260806 20:35:15.507661 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:15.508697 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:15.508961 14627 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:15.509510 14625 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 20:35:15.509893 14625 pending_rounds.cc:76] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 1
I20260806 20:35:15.510102 14625 pending_rounds.cc:97] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 20:35:15.510324 14625 raft_consensus.cc:2866] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:15.510571 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.493470 16132 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:15.514349 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:15.510162 15153 leader_election.cc:334] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.877s (SENT)
I20260806 20:35:15.513119 14633 raft_consensus.cc:1261] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:15.514962 15184 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:15.511785 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.515998 15153 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.510556 14624 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:15.465888 16151 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.505887 16081 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:15.511935 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:15.527046 14629 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:15.528785 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:15.542729 15187 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:15.530129 16043 raft_consensus.cc:3032] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:15.530737 16122 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.531090 14628 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
W20260806 20:35:15.531180 14444 leader_election.cc:334] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.631s (SENT)
I20260806 20:35:15.527034 16167 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.532377 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.521812 16132 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.532893 16191 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.536532 16067 raft_consensus.cc:3032] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:15.537930 16110 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:15.514546 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:15.539639 14627 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:15.547155 14625 raft_consensus.cc:1261] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 20:35:15.571606 14625 pending_rounds.cc:76] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 1
I20260806 20:35:15.571888 14625 pending_rounds.cc:97] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20260806 20:35:15.571771 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 169053us
I20260806 20:35:15.572283 14625 raft_consensus.cc:2866] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:15.522483 16182 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.522686 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.524135 16193 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.580777 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:15.564675 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:15.566160 14444 leader_election.cc:334] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.696s (SENT)
I20260806 20:35:15.566452 16187 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:15.567553 14553 raft_consensus.cc:1261] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:15.564303 15183 raft_consensus.cc:1226] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:15.547197 16043 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:15.588016 16204 log.cc:924] Time spent T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260806 20:35:15.588506 15185 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 20:35:15.588676 16179 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:15.592105 15184 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:15.593590 15183 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:15.561100 16067 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:15.597609 14518 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:15.667562 16203 log.cc:924] Time spent T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260806 20:35:15.670936 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:15.669920 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 103811us
W20260806 20:35:15.666466 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 263508us
I20260806 20:35:15.676277 14521 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:15.672631 14444 leader_election.cc:334] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.723s (SENT)
W20260806 20:35:15.677505 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 77314.2us
I20260806 20:35:15.678825 16122 consensus_queue.cc:1046] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:15.678997 16211 consensus_queue.cc:1046] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 20:35:15.680217 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:15.681454 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 94374.9us
I20260806 20:35:15.671564 15837 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.680577 15184 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 20:35:15.669492 16130 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:15.685690 16049 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:15.694940 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.686714 16205 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:15.687122 16172 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.555s user 0.008s sys 0.006s
I20260806 20:35:15.691514 14595 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:15.686348 16214 log.cc:924] Time spent T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260806 20:35:15.695319 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:15.703557 15187 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:15.707801 14518 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.696799 15184 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:15.713640 14595 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.714828 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.586833 16187 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.724427 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:15.701649 15185 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:15.727988 15184 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:35:15.737937 15153 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.723791 16130 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.721045 16077 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.720991 16049 raft_consensus.cc:686] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:15.741297 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:15.727123 16205 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:15.744253 15153 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.747231 15184 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:15.748178 16191 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.748816 16191 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.740546 15181 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:15.734822 16043 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.715509 16172 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.728013 16110 consensus_queue.cc:1046] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.019s
I20260806 20:35:15.753031 15837 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:15.756882 16208 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:15.761353 14477 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:15.743366 16125 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.749644 15153 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.755199 16132 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:15.770426 16220 log.cc:924] Time spent T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260806 20:35:15.755505 16205 raft_consensus.cc:686] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:15.770519 15837 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.751474 16077 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:15.751086 16130 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.728689 16067 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.775413 14549 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:15.774277 16211 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:15.766883 15918 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:15.780033 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 102939us
W20260806 20:35:15.787163 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 63641.2us
I20260806 20:35:15.788699 14521 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.789783 15153 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.773974 16208 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.791746 14596 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.780194 16127 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.771389 16049 consensus_queue.cc:237] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.786907 16043 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:15.794121 16067 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:15.796557 16093 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.806531 16187 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:15.801167 16130 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:15.809125 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.788888 16110 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260806 20:35:15.791383 15918 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.813957 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.799160 16217 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.824342 14625 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 2: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:15.811033 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:15.787251 16077 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.830502 15153 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.830256 16205 consensus_queue.cc:237] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.832370 16077 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.827327 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.835393 16215 raft_consensus.cc:3032] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:15.835916 16215 raft_consensus.cc:729] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:15.835938 14624 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:15.839452 15837 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.835333 15918 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:15.841524 15183 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:15.838944 16172 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:15.837353 16187 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.839179 16215 consensus_queue.cc:260] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.846609 16215 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:15.846860 16221 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.069s user 0.000s sys 0.003s
I20260806 20:35:15.827991 16093 consensus_queue.cc:1035] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260806 20:35:15.824370 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 109623us
I20260806 20:35:15.850613 16208 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.850986 14595 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:15.851207 16226 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:15.851815 16226 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.815465 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.789003 16211 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.858682 14625 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:15.862365 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:15.863518 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.864837 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.864303 14626 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:15.867697 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.868358 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.854184 15153 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:15.855505 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.857306 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.874085 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:15.874699 14630 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:15.865625 14632 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:15.876143 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:15.880880 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.883232 14551 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:15.884826 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.886209 14595 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:15.889963 14551 raft_consensus.cc:1226] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:15.858361 15961 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.892804 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.892902 14551 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 20:35:15.897187 14551 pending_rounds.cc:76] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 5
I20260806 20:35:15.897486 14551 pending_rounds.cc:97] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 20:35:15.897760 14551 raft_consensus.cc:2866] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:15.894420 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:15.900158 14553 pending_rounds.cc:76] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 5
I20260806 20:35:15.900511 14553 pending_rounds.cc:97] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 20:35:15.900758 14553 raft_consensus.cc:2866] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:15.903263 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:15.903959 14595 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:15.904843 14626 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:15.882591 15996 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.910982 15151 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.913900 15151 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.880848 16208 consensus_queue.cc:1046] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260806 20:35:15.916146 16226 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:15.892628 16132 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:15.916806 16226 raft_consensus.cc:3032] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:15.901264 15178 raft_consensus.cc:2379] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:15.896149 16125 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:15.883095 16081 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:15.917215 16226 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:15.929278 14521 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:15.932788 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.917402 15961 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:15.936300 15961 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:15.936636 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.937022 15961 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:15.938180 16222 consensus_queue.cc:1046] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:15.891873 16130 consensus_queue.cc:1046] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260806 20:35:15.917450 16182 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.917934 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.922109 16132 raft_consensus.cc:686] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:15.945067 14521 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.921422 15996 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:15.925119 16187 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:15.948930 14550 raft_consensus.cc:1226] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:15.917336 16193 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 20:35:15.944103 16208 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:15.939393 16125 raft_consensus.cc:686] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:15.974228 16225 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:15.974494 16228 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:15.975250 16230 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:15.975930 15953 raft_consensus.cc:3032] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:15.978389 16208 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:15.982455 16132 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:15.984474 16211 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:16.002081 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 57056.7us
I20260806 20:35:16.062983 14521 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:16.051294 14595 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:15.986610 16125 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.059147 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:15.921142 16217 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
W20260806 20:35:16.067444 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 372807us
I20260806 20:35:16.072000 14521 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:15.960615 15996 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.085613 16187 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:16.085999 15996 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:16.086117 16187 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:16.069110 15182 raft_consensus.cc:1226] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:16.050402 15953 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:16.082866 16253 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:16.060663 16208 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:16.092479 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:16.093509 14627 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:16.080777 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:16.014047 16209 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:16.100734 14630 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:16.064944 16225 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:16.112361 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:16.155756 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 483157us
W20260806 20:35:16.158603 14444 leader_election.cc:334] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.895s (SENT)
I20260806 20:35:16.159369 14444 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:16.084057 15961 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.086614 15996 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:16.103755 16093 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:15.975265 16226 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.102042 16157 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:16.109694 16049 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:16.206959 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 138689us
W20260806 20:35:16.198437 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 523270us
I20260806 20:35:16.176549 16031 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:16.199239 16093 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:16.254828 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 203644us
W20260806 20:35:16.260123 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 101571us
I20260806 20:35:16.157148 16253 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.242058 16168 consensus_queue.cc:1046] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260806 20:35:16.297199 16077 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:16.297863 16077 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.258737 15918 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:16.300081 16077 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:16.299791 15918 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.226380 16179 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:16.312590 14444 leader_election.cc:334] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.766s (SENT)
I20260806 20:35:16.325337 14444 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:16.246161 16211 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:16.253527 16093 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:16.228395 15821 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20260806 20:35:16.330081 16065 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:16.340273 16249 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
W20260806 20:35:16.341285 16245 log.cc:924] Time spent T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20260806 20:35:16.224355 16157 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.350965 16253 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:16.355816 15961 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:16.155944 16225 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.361033 15918 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:16.362504 16129 raft_consensus.cc:3032] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:16.366186 16187 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.259085 16081 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:16.332005 16043 consensus_meta.cc:287] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.528s user 0.011s sys 0.001s
I20260806 20:35:16.372872 16227 raft_consensus.cc:3032] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:16.248261 16151 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.120s
W20260806 20:35:16.375707 16215 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.529s user 0.005s sys 0.010s
W20260806 20:35:16.377779 16067 consensus_meta.cc:287] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.562s user 0.010s sys 0.000s
I20260806 20:35:16.244850 15953 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.368835 16211 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:16.395143 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 82608.3us
W20260806 20:35:16.439921 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 326331us
I20260806 20:35:16.403105 16129 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:16.446995 14521 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:16.408591 16227 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:16.459869 14521 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:16.494920 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:16.496779 14630 raft_consensus.cc:1261] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:16.496783 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:16.498242 14625 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:16.516043 14626 raft_consensus.cc:1261] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:16.538576 14521 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:16.496126 16067 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:16.543550 16232 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:16.543938 16232 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:16.544317 16232 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:16.544216 16049 consensus_queue.cc:1035] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:16.545367 14553 raft_consensus.cc:1261] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:16.526896 16207 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:16.548026 16187 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:16.513046 15837 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:16.549185 15837 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:16.549683 15837 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:16.550019 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:16.550858 14625 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:16.554984 16157 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:16.529867 16200 consensus_queue.cc:1035] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260806 20:35:16.548043 16207 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.512480 15961 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:16.596007 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 373773us
I20260806 20:35:16.614288 16253 consensus_queue.cc:1035] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:16.600543 16255 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:16.633736 16186 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:35:16.626091 15961 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.603596 16240 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:16.641101 16240 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.603353 16229 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:16.646306 16229 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.656351 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1260) took 1984 ms. Trace:
I20260806 20:35:16.656920 14552 rpcz_store.cc:276] 0806 20:35:14.671936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:14.672039 (+ 103us) service_pool.cc:227] Handling call
0806 20:35:14.680313 (+ 8274us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:14.749957 (+ 69644us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:14.749987 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:14.761014 (+ 11027us) log.cc:842] Serialized 26 byte log entry
0806 20:35:15.022384 (+261370us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b18002ef480. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a411 00007fb9b34c75cf
0806 20:35:15.184228 (+161844us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:15.184269 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:16.655940 (+1471671us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b54000861d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:16.656005 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:16.656076 (+ 71us) raft_consensus.cc:1671] finished
0806 20:35:16.656093 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:16.656252 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1739,"raft.queue_time_us":406,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":10961,"spinlock_wait_cycles":152639616,"thread_start_us":161672,"threads_started":1,"wal-append.queue_time_us":426159,"wal-append.run_cpu_time_us":2075,"wal-append.run_wall_time_us":1171779}
I20260806 20:35:16.660055 14632 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:16.530817 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:16.660076 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 402771us
I20260806 20:35:16.666168 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:16.666143 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 438691us
I20260806 20:35:16.673683 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:16.605450 16067 raft_consensus.cc:686] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:16.604878 16200 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:16.686739 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 81869.6us
W20260806 20:35:16.688211 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 7f41753baced49a1997940dfc28b2e00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:16.622757 16227 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.703459 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:16.705077 15151 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.004s sys 0.019s
W20260806 20:35:16.705400 15151 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.642s user 0.004s sys 0.019s
I20260806 20:35:16.707324 14518 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:16.707762 14478 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:35:16.709605 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:16.712167 15187 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:16.719832 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 113727us
I20260806 20:35:16.674875 16151 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
W20260806 20:35:16.710283 16132 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.022s sys 0.052s
I20260806 20:35:16.722204 14625 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:16.695961 15722 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:16.728428 15153 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:16.682018 15996 consensus_queue.cc:1046] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260806 20:35:16.593129 16129 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.704751 16178 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:16.830173 14550 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 20:35:16.837543 16254 log.cc:924] Time spent T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260806 20:35:16.693908 16200 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.875034 14469 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:16.838016 15722 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.876832 14549 raft_consensus.cc:2396] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:16.877008 16233 consensus_queue.cc:1046] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.157s
I20260806 20:35:16.877372 16186 consensus_queue.cc:1046] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:16.878090 15722 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:16.878218 16258 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:16.878698 16258 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.880618 16265 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:16.881192 16265 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.883292 16265 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:16.884151 16093 consensus_meta.cc:287] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.502s user 0.011s sys 0.004s
W20260806 20:35:16.884310 16256 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.188s user 0.000s sys 0.000s
I20260806 20:35:16.876852 16200 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:16.838433 14475 raft_consensus.cc:1261] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:16.894811 15961 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:16.897624 16065 consensus_meta.cc:287] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.544s user 0.010s sys 0.000s
I20260806 20:35:16.898164 16065 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:16.900401 16208 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.702s user 0.010s sys 0.007s
I20260806 20:35:16.900583 14471 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:16.900900 16208 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:16.720929 16173 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:16.721761 16209 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:16.730944 16132 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.023s sys 0.053s
I20260806 20:35:16.682706 16255 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.906224 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:16.947865 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 144237us
I20260806 20:35:16.957998 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:16.958741 14555 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:16.958873 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:16.962617 14551 raft_consensus.cc:1226] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260806 20:35:16.958789 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 230472us
I20260806 20:35:16.970260 14596 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:16.971172 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:16.877979 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:16.897677 14546 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260806 20:35:16.975868 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 267007us
I20260806 20:35:16.902173 16225 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:16.978663 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 248043us
I20260806 20:35:17.009009 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:17.009222 16168 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:16.941123 16136 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.015717 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:17.016713 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:16.926007 16240 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:16.902803 16178 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:16.959233 16200 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.036111 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.002511 16132 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:16.978158 16230 raft_consensus.cc:3032] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:17.003399 16275 raft_consensus.cc:3032] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:16.922974 16093 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.021328 16173 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.022594 16209 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.025111 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:17.153443 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.028216 16172 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:16.907361 16067 consensus_queue.cc:237] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:17.200635 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47850.3us
W20260806 20:35:17.167776 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 142001us
I20260806 20:35:17.035369 15918 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.028898 14631 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:17.035785 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.035460 15722 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:17.117944 16230 raft_consensus.cc:729] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:17.090327 16132 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.077214 16168 raft_consensus.cc:686] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:17.028209 16225 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.176s
I20260806 20:35:17.137753 16275 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:17.152976 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.155194 15957 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.088063 15996 consensus_queue.cc:1035] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.167s
I20260806 20:35:17.211107 16049 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.037984 16136 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.089951 16200 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.274272 14444 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.168746 16229 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.233204 16132 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:17.250679 15918 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.214529 16211 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:17.273402 16266 log.cc:924] Time spent T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.245s user 0.000s sys 0.001s
I20260806 20:35:17.232834 16258 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:17.307715 14444 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.297088 16281 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:17.297573 16255 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:17.284402 16244 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.311041 16244 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.311704 16208 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.231148 16209 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.313367 16244 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:17.273957 16230 consensus_queue.cc:260] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.315109 16230 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:17.316320 16093 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.301367 16178 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.316691 14369 catalog_manager.cc:5495] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 2, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 20:35:17.288157 16232 consensus_meta.cc:287] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.743s user 0.016s sys 0.016s
I20260806 20:35:17.317739 14444 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:17.313388 14370 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56582 (request call id 138) took 11650 ms (client timeout 15000 ms). Trace:
I20260806 20:35:17.318173 16232 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.319545 14553 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:17.319751 16281 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:17.320174 16281 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:17.273124 16051 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.315040 16209 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.321655 16051 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.308619 16269 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:17.322808 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:17.252358 14479 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
W20260806 20:35:17.318743 14370 rpcz_store.cc:269] 0806 20:35:05.662663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:05.663145 (+ 482us) service_pool.cc:227] Handling call
0806 20:35:17.313288 (+11650143us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:14.066267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:14.085218 (+ 18951us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:14.085275 (+ 57us) write_op.cc:435] Acquired schema lock
0806 20:35:14.085304 (+ 29us) tablet.cc:660] Decoding operations
0806 20:35:14.085777 (+ 473us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:14.085863 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:14.085912 (+ 49us) tablet.cc:683] Acquiring locks for 2 operations
0806 20:35:14.086467 (+ 555us) tablet.cc:699] Row locks acquired
0806 20:35:14.086496 (+ 29us) write_op.cc:260] PREPARE: finished
0806 20:35:14.086663 (+ 167us) write_op.cc:270] Start()
0806 20:35:14.086807 (+ 144us) write_op.cc:276] Timestamp: P: 1786048514086605 usec, L: 0
0806 20:35:14.086838 (+ 31us) op_driver.cc:331] REPLICATION: starting
0806 20:35:14.139206 (+ 52368us) log.cc:842] Serialized 925 byte log entry
0806 20:35:14.362233 (+223027us) op_driver.cc:444] REPLICATION: finished
0806 20:35:15.023580 (+661347us) spinlock_profiling.cc:243] Waited 59.9 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b05ad5fe 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b6f03410
0806 20:35:15.076067 (+ 52487us) write_op.cc:301] APPLY: starting
0806 20:35:15.076173 (+ 106us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:15.076303 (+ 130us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:15.076337 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:15.076829 (+ 492us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:15.077096 (+ 267us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 20:35:15.077161 (+ 65us) write_op.cc:312] APPLY: finished
0806 20:35:15.077431 (+ 270us) log.cc:842] Serialized 29 byte log entry
0806 20:35:15.154433 (+ 77002us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:15.154548 (+ 115us) write_op.cc:454] Released schema lock
0806 20:35:15.161348 (+ 6800us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:15.191274 (+ 29926us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:15.193677 (+ 2403us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":21084,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":651744,"apply.run_cpu_time_us":2786,"apply.run_wall_time_us":147439,"num_ops":2,"prepare.queue_time_us":235610,"prepare.run_cpu_time_us":19020,"prepare.run_wall_time_us":226168,"raft.queue_time_us":9042,"raft.run_cpu_time_us":18758,"raft.run_wall_time_us":829444,"replication_time_us":248575,"spinlock_wait_cycles":137764608,"thread_start_us":877131,"threads_started":4,"wal-append.queue_time_us":92964,"wal-append.run_cpu_time_us":12216,"wal-append.run_wall_time_us":1789533}]]}
I20260806 20:35:17.323655 16051 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.324219 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.325521 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:17.201308 16173 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.320639 16281 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:17.276329 15957 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.328178 16173 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:17.288237 16261 raft_consensus.cc:3032] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:17.333158 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:17.347999 14444 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.330494 16232 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.277431 14469 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
W20260806 20:35:17.306027 15837 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.756s user 0.008s sys 0.011s
I20260806 20:35:17.329838 15957 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.219131 16172 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.329015 16275 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.319078 14369 catalog_manager.cc:5495] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 2, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:17.324245 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.354082 16051 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:17.365990 16226 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:17.367607 16136 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.350096 16200 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.369328 15918 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.276780 16049 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.370548 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.328440 16227 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:17.353088 16168 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.372046 15837 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.352718 15953 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:17.345239 16261 raft_consensus.cc:729] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:17.339501 14470 raft_consensus.cc:1226] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:17.295341 16207 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.402037 14369 catalog_manager.cc:5495] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 2, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260806 20:35:17.407958 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 60046.2us
I20260806 20:35:17.423906 14444 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
W20260806 20:35:17.410235 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 459129us
I20260806 20:35:17.395315 16275 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:17.394390 16172 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.435279 16132 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.437397 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.379794 16157 consensus_queue.cc:1046] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260806 20:35:17.380041 16051 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:17.403398 15821 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.033s sys 0.066s
I20260806 20:35:17.460213 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.485411 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.485586 16049 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.486801 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.437820 16261 consensus_queue.cc:260] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:17.490476 15821 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.600s user 0.033s sys 0.067s
I20260806 20:35:17.446246 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:17.462056 16028 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:17.480191 16241 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.496152 16241 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.496969 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.497366 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.497908 14473 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:17.483362 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:17.459656 16262 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:17.488199 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:17.479249 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 55415.8us
I20260806 20:35:17.513569 16173 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.514369 16173 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.477908 15837 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.521982 16173 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.498705 16241 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.525110 16132 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:17.526896 16270 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.722s user 0.000s sys 0.004s
I20260806 20:35:17.533867 14473 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:17.491247 16276 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.336s user 0.000s sys 0.002s
I20260806 20:35:17.494714 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 196) took 1383 ms. Trace:
I20260806 20:35:17.541388 16261 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:17.547601 16264 log.cc:924] Time spent T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.836s user 0.000s sys 0.011s
I20260806 20:35:17.544046 16129 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:17.549280 15187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 419) took 1862 ms (client timeout 1896 ms). Trace:
I20260806 20:35:17.553140 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.404841 14474 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:17.572680 14470 raft_consensus.cc:1261] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:17.577455 16207 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:17.578329 16049 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.579356 16263 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:35:17.571919 14551 rpcz_store.cc:276] 0806 20:35:16.111632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:16.114007 (+ 2375us) service_pool.cc:227] Handling call
0806 20:35:16.114376 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:16.963144 (+848768us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:17.494401 (+531257us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b54000b36d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:17.494642 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1950175616}
I20260806 20:35:17.491612 16051 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:17.589473 15187 rpcz_store.cc:269] 0806 20:35:15.687278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:15.724022 (+ 36744us) service_pool.cc:227] Handling call
0806 20:35:17.549023 (+1825001us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b54006905d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:17.549155 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1621102464}
I20260806 20:35:17.507404 16028 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:17.588680 14629 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.016s sys 0.057s
I20260806 20:35:17.450047 16281 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:17.609084 14629 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s user 0.016s sys 0.058s
I20260806 20:35:17.588814 16077 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.586243 16200 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.613644 16077 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.614017 16200 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.609985 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:17.610822 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 102870us
I20260806 20:35:17.617707 14444 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.611896 16028 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:17.607561 14475 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:17.605177 16207 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.643728 14444 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:17.642463 16031 raft_consensus.cc:3032] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:17.663115 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 191) took 1760 ms. Trace:
I20260806 20:35:17.624969 16157 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:17.610390 16049 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:17.705277 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 76316.6us
I20260806 20:35:17.706418 14444 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:17.709731 15153 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.034s sys 0.031s
W20260806 20:35:17.710136 15153 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.034s sys 0.031s
I20260806 20:35:17.715603 14444 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:17.653407 14474 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:17.664161 16031 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:17.700142 16200 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.720105 16191 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:17.719100 16077 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.725219 16157 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:17.734537 16031 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.736013 16157 raft_consensus.cc:686] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:17.736498 16031 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:17.721817 16225 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.737552 16157 consensus_queue.cc:237] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.739965 16257 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.740774 16257 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.726078 16043 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:17.745604 16043 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.733275 16191 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.749490 16281 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.740113 16225 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.733604 16207 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:17.765918 14479 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
W20260806 20:35:17.775504 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 49801us
I20260806 20:35:17.776525 14444 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:17.781092 15151 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.017s sys 0.039s
W20260806 20:35:17.781500 15151 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.880s user 0.017s sys 0.039s
W20260806 20:35:17.781838 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 940573us
I20260806 20:35:17.779978 16207 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.839814 16207 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.670431 14547 rpcz_store.cc:276] 0806 20:35:15.902354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:15.905230 (+ 2876us) service_pool.cc:227] Handling call
0806 20:35:15.928703 (+ 23473us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:15.987751 (+ 59048us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:16.012484 (+ 24733us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:16.122770 (+110286us) log.cc:842] Serialized 26 byte log entry
0806 20:35:16.463343 (+340573us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:16.463420 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:16.463567 (+ 147us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:17.546775 (+1083208us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:17.629784 (+ 83009us) raft_consensus.cc:1671] finished
0806 20:35:17.629826 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:17.662977 (+ 33151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11715,"raft.queue_time_us":13434,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":71201,"spinlock_wait_cycles":405755136,"thread_start_us":225950,"threads_started":1,"wal-append.queue_time_us":479407}
W20260806 20:35:17.828003 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 51428.5us
I20260806 20:35:17.845968 16043 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.846676 14444 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.847723 16179 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:17.848558 14444 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.850210 14444 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:17.781795 16281 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:17.816627 16217 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.836100 16225 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.850961 16184 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:17.861943 16179 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:17.873556 16179 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:17.874315 16179 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:17.867614 16207 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:17.870117 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 80861.2us
I20260806 20:35:17.880470 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:17.894454 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 44302.2us
I20260806 20:35:17.894927 16217 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.895359 16225 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:17.875975 16043 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20260806 20:35:17.901741 16110 consensus_queue.cc:1046] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:17.894979 14474 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:17.904749 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.907141 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.911424 16217 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.865494 16191 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.928972 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.929634 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:17.929804 14479 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:17.930495 14551 raft_consensus.cc:2396] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:35:17.931191 16127 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:17.931918 16127 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:17.895756 14444 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:17.933449 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:17.933961 16184 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:17.934398 14444 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:17.935922 16184 raft_consensus.cc:2741] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:17.936266 16281 consensus_queue.cc:1035] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:17.938817 16127 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.940223 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:17.945524 16257 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.948750 16262 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:17.949546 14547 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:17.924247 16051 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:17.967191 15178 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:17.967898 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:17.968045 15183 raft_consensus.cc:1226] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:17.925529 16049 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:17.968899 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:17.974025 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:17.996313 14548 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:17.973452 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:17.997321 14553 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:18.007498 15151 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:18.019733 14471 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.000s sys 0.033s
W20260806 20:35:18.032578 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 96689.8us
W20260806 20:35:18.020385 14471 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.012s user 0.000s sys 0.034s
I20260806 20:35:18.056387 16262 consensus_queue.cc:1035] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:18.056953 14444 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:18.059051 15151 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:18.034121 14469 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:35:18.087569 15151 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:18.059437 16206 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.099s user 0.021s sys 0.014s
W20260806 20:35:18.091157 16206 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.145s user 0.021s sys 0.014s
I20260806 20:35:18.091802 16286 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:18.092412 16286 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:18.092855 16286 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:18.089290 16217 consensus_queue.cc:1046] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260806 20:35:18.099058 16217 raft_consensus.cc:3032] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:18.099608 16217 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:18.089031 14473 raft_consensus.cc:1261] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:18.115173 15953 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:18.113193 16184 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:18.123524 14477 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.095s user 0.016s sys 0.020s
I20260806 20:35:18.031437 16028 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:18.123911 14477 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.095s user 0.016s sys 0.020s
I20260806 20:35:18.095865 16257 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:18.114189 16110 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:18.139075 14553 raft_consensus.cc:1226] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260806 20:35:18.135468 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 78718.2us
W20260806 20:35:18.124974 16065 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.017s sys 0.029s
W20260806 20:35:18.142668 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.000s sys 0.029s
I20260806 20:35:18.149657 14476 raft_consensus.cc:1226] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
W20260806 20:35:18.151636 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.974s user 0.000s sys 0.030s
W20260806 20:35:18.136694 15946 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.112s user 0.016s sys 0.019s
W20260806 20:35:18.163692 14478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.006s sys 0.005s
I20260806 20:35:18.191543 16297 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:18.192229 16297 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.193778 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:18.194007 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.195672 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.197783 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.198021 16297 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:18.137594 16257 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:18.202085 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 44034.5us
W20260806 20:35:18.191071 15946 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.181s user 0.017s sys 0.019s
W20260806 20:35:18.205152 14478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.006s sys 0.005s
W20260806 20:35:18.151005 16065 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.014s user 0.018s sys 0.029s
I20260806 20:35:18.213109 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:18.137802 16110 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:18.216545 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.221879 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 161) took 1510 ms. Trace:
I20260806 20:35:18.222354 14476 rpcz_store.cc:276] 0806 20:35:16.699279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:16.699412 (+ 133us) service_pool.cc:227] Handling call
0806 20:35:17.547610 (+848198us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:17.547858 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:18.151320 (+603462us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:18.209861 (+ 58541us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b54003c4454. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:18.210099 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1388654336}
I20260806 20:35:18.214051 16184 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260806 20:35:18.234380 16179 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:18.143857 16217 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.243367 16179 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:18.189710 15953 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:18.200155 16287 consensus_queue.cc:1046] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:18.262331 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.156358 16261 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:18.263104 15184 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:18.223696 16110 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:18.271981 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.224712 16051 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:18.285315 14555 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.001s user 0.030s sys 0.033s
W20260806 20:35:18.287864 14555 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.004s user 0.030s sys 0.033s
I20260806 20:35:18.291894 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.293792 14518 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:18.260251 16127 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:18.220690 16262 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260806 20:35:18.261996 16217 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:18.310395 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:18.313688 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:18.316704 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.144s user 0.027s sys 0.055s
W20260806 20:35:18.317245 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.144s user 0.027s sys 0.056s
W20260806 20:35:18.314592 15181 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.285s user 0.008s sys 0.039s
W20260806 20:35:18.318149 15181 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.303s user 0.008s sys 0.039s
I20260806 20:35:18.298797 16179 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:18.268587 16065 consensus_queue.cc:1035] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
W20260806 20:35:18.321041 14552 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.154s user 0.022s sys 0.022s
I20260806 20:35:18.305051 16297 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:18.344551 14552 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.199s user 0.022s sys 0.022s
I20260806 20:35:18.307822 16206 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:18.269714 16261 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:18.283583 16184 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260806 20:35:18.267251 15953 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.386757 16285 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:18.387578 16285 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.283978 16191 consensus_queue.cc:1046] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 20:35:18.310637 16127 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:18.406739 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 60306.7us
I20260806 20:35:18.406653 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:18.346442 16206 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:18.385219 16241 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:18.416250 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:18.433773 14443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.218s user 0.000s sys 0.041s
W20260806 20:35:18.434237 14443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.219s user 0.000s sys 0.041s
I20260806 20:35:18.435060 14595 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:18.403873 16127 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:18.414706 16206 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:18.383297 16297 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.429183 14469 raft_consensus.cc:3032] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:18.444121 16028 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:18.462051 16257 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:18.485852 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1288) took 1400 ms. Trace:
I20260806 20:35:18.480697 14553 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:18.433463 16241 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.468797 14469 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:18.492800 16193 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.204s user 0.010s sys 0.030s
I20260806 20:35:18.486384 14555 rpcz_store.cc:276] 0806 20:35:17.085748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.091074 (+ 5326us) service_pool.cc:227] Handling call
0806 20:35:17.091464 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:17.283271 (+191807us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:17.283715 (+ 444us) log.cc:842] Serialized 14 byte log entry
0806 20:35:18.293544 (+1009829us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:18.293607 (+ 63us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:18.293648 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:18.381765 (+ 88117us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b18002ef9c0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b0598f96 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9
0806 20:35:18.410056 (+ 28291us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54001397d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:18.410119 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:18.485785 (+ 75666us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4930,"prepare.queue_time_us":4874,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":5094,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":5214,"spinlock_wait_cycles":467856512,"thread_start_us":1004403,"threads_started":1}
W20260806 20:35:18.482959 14521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.580s (SENT) [suppressed 52 similar messages]
W20260806 20:35:18.487936 16168 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.041s user 0.031s sys 0.025s
W20260806 20:35:18.506891 15186 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.169s user 0.019s sys 0.031s
W20260806 20:35:18.510476 15186 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.172s user 0.019s sys 0.031s
W20260806 20:35:18.514599 14441 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.015s sys 0.024s
W20260806 20:35:18.527760 14441 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.973s user 0.015s sys 0.024s
I20260806 20:35:18.529263 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:18.506666 16193 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.222s user 0.010s sys 0.030s
I20260806 20:35:18.499369 16285 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:18.509603 16168 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.063s user 0.031s sys 0.025s
W20260806 20:35:18.528692 14598 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.121s user 0.004s sys 0.038s
W20260806 20:35:18.536813 14598 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.365s user 0.004s sys 0.039s
W20260806 20:35:18.509690 16166 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.635s user 0.004s sys 0.044s
W20260806 20:35:18.522859 16187 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.141s user 0.010s sys 0.026s
I20260806 20:35:18.530766 16225 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:18.546453 16166 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.670s user 0.004s sys 0.044s
W20260806 20:35:18.545881 16187 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.172s user 0.010s sys 0.026s
W20260806 20:35:18.550572 16031 consensus_meta.cc:287] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.814s user 0.010s sys 0.007s
I20260806 20:35:18.546535 15953 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:18.542917 15182 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.533s user 0.013s sys 0.031s
I20260806 20:35:18.553100 15180 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:35:18.553759 16261 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:18.543798 16285 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:18.549502 16067 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.084s user 0.019s sys 0.026s
W20260806 20:35:18.529613 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 975614us
I20260806 20:35:18.550572 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.621145 14479 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:18.543586 14469 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:18.647514 14369 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.021s sys 0.060s
W20260806 20:35:18.651669 14369 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.043s user 0.022s sys 0.060s
W20260806 20:35:18.625602 14546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.270s user 0.014s sys 0.043s
W20260806 20:35:18.620777 16151 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.322s user 0.032s sys 0.050s
W20260806 20:35:18.622522 15182 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.613s user 0.013s sys 0.032s
W20260806 20:35:18.648245 15821 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.156s user 0.015s sys 0.066s
W20260806 20:35:18.678011 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 148845us
W20260806 20:35:18.650724 16067 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.186s user 0.021s sys 0.026s
W20260806 20:35:18.677929 16151 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.419s user 0.033s sys 0.051s
I20260806 20:35:18.716547 16077 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:18.649112 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:18.717398 16077 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:18.717995 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.718152 16295 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:18.719332 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.718950 16295 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:18.719987 14473 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:18.706561 16290 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260806 20:35:18.722571 16241 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:18.686316 15821 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.195s user 0.016s sys 0.066s
I20260806 20:35:18.655488 16285 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.724999 16285 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:18.725719 16285 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:18.726115 16285 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:18.726531 16285 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:18.658203 16273 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.041s user 0.013s sys 0.035s
I20260806 20:35:18.624657 15953 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:18.674062 14546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.322s user 0.014s sys 0.043s
I20260806 20:35:18.714046 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:18.680980 16261 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:18.676759 14469 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:18.678057 16167 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.478s user 0.030s sys 0.026s
W20260806 20:35:18.730877 16273 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.114s user 0.013s sys 0.038s
I20260806 20:35:18.641675 16287 raft_consensus.cc:3032] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:18.722465 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.780418 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.782276 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.708709 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:18.648470 16297 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:18.676198 16127 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:18.681453 16049 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:18.715104 14518 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:18.777279 16287 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:18.769291 16207 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:18.771760 16293 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
W20260806 20:35:18.773857 16167 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.607s user 0.030s sys 0.026s
W20260806 20:35:18.797192 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 82166.6us
I20260806 20:35:18.717957 16077 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:18.784253 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.804138 14518 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:18.803025 16207 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:18.801743 16067 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:18.774982 16261 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.815528 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.796566 16127 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.772848 15953 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.810129 16295 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:18.817847 16077 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:18.817329 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.793717 16049 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:18.822901 16067 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:18.832705 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:18.833377 16217 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.517s user 0.009s sys 0.002s
I20260806 20:35:18.834463 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.835315 14473 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:18.839208 16049 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:18.839869 16110 consensus_meta.cc:287] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.533s user 0.007s sys 0.002s
I20260806 20:35:18.841655 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.843187 14479 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:18.844275 14476 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:18.844321 15150 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:18.825554 16287 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.853953 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:18.872961 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.836372 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.870956 16287 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:18.893889 14475 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.141s user 0.021s sys 0.024s
W20260806 20:35:18.926293 14519 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.746s user 0.028s sys 0.062s
W20260806 20:35:18.926707 14519 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.746s user 0.028s sys 0.062s
I20260806 20:35:18.868924 16110 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:18.934720 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.936966 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:18.924527 14475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.169s user 0.021s sys 0.024s
I20260806 20:35:18.920835 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.941498 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:18.946990 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:18.947477 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:18.948010 14553 raft_consensus.cc:2396] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 20:35:18.942443 14470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.215s user 0.031s sys 0.053s
W20260806 20:35:18.941180 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.787s user 0.011s sys 0.049s
W20260806 20:35:18.949169 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.795s user 0.011s sys 0.049s
I20260806 20:35:18.931062 16067 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:18.888967 16157 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.059s user 0.011s sys 0.030s
W20260806 20:35:19.008633 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 57428.7us
W20260806 20:35:18.948824 14470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.222s user 0.031s sys 0.053s
W20260806 20:35:18.949657 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 1.79529e+06us
I20260806 20:35:18.954977 14476 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:18.920351 16206 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.024545 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:19.023069 15181 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.015s sys 0.006s
I20260806 20:35:19.026121 15183 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:35:19.021262 16166 consensus_queue.cc:1046] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
W20260806 20:35:19.010272 16157 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.180s user 0.011s sys 0.030s
I20260806 20:35:19.010401 16127 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.023080 16067 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.029799 16295 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:19.041110 16286 consensus_meta.cc:287] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.948s user 0.000s sys 0.015s
I20260806 20:35:19.067193 16295 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.067396 16286 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.041229 16031 raft_consensus.cc:3032] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:19.068943 16031 raft_consensus.cc:729] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:19.069284 16295 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.069806 16031 consensus_queue.cc:260] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.071053 16031 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:19.041326 16261 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:19.073020 16261 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:19.073472 16261 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.075608 16261 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:19.050988 16207 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:19.042361 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:19.077834 14476 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:19.081177 15953 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:19.062420 16110 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:19.032109 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.032356 16296 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:19.031842 15181 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.015s sys 0.007s
I20260806 20:35:19.076251 16261 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.034749 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:19.040868 16294 log.cc:924] Time spent T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260806 20:35:19.104067 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:19.079720 16122 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:19.063198 14473 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:19.069825 16295 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:19.065623 16067 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:19.125063 14479 raft_consensus.cc:1226] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:19.110280 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1289) took 2022 ms. Trace:
W20260806 20:35:19.131445 14554 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.135s user 0.023s sys 0.047s
I20260806 20:35:19.114836 15953 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:19.126302 16273 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:19.131080 14550 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.077s user 0.018s sys 0.053s
W20260806 20:35:19.136168 14550 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.082s user 0.018s sys 0.053s
W20260806 20:35:19.132046 14554 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.144s user 0.024s sys 0.047s
I20260806 20:35:19.081223 16286 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:19.118960 16263 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:19.115062 16261 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.121227 16122 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.113816 16296 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.125571 16295 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.144567 15953 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:19.180197 14471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.138s user 0.005s sys 0.056s
W20260806 20:35:19.180585 14471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.138s user 0.006s sys 0.056s
W20260806 20:35:19.178493 14549 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.206s user 0.011s sys 0.046s
W20260806 20:35:19.186139 14549 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.214s user 0.011s sys 0.046s
I20260806 20:35:19.144586 16273 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:19.187268 14551 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.215s user 0.018s sys 0.095s
I20260806 20:35:19.205387 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1482) took 2242 ms. Trace:
I20260806 20:35:19.152380 16263 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.213186 14471 rpcz_store.cc:276] 0806 20:35:16.941976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:16.942136 (+ 160us) service_pool.cc:227] Handling call
0806 20:35:16.942472 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:16.942669 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:18.042372 (+1099703us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.183207 (+1140835us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:19.184322 (+ 1115us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:19.184381 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.184500 (+ 119us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.184634 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":364495,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":29,"raft.queue_time_us":110,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":192,"thread_start_us":2171807,"threads_started":2}
W20260806 20:35:19.212932 14551 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.241s user 0.019s sys 0.096s
W20260806 20:35:19.219269 14444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.042s sys 0.072s
I20260806 20:35:19.185931 16186 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:19.177253 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:19.210904 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 210) took 1240 ms. Trace:
I20260806 20:35:19.212711 15178 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:19.221150 14444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.976s user 0.042s sys 0.072s
W20260806 20:35:19.228967 14477 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.092s user 0.040s sys 0.054s
W20260806 20:35:19.231889 14477 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.095s user 0.040s sys 0.054s
I20260806 20:35:19.233143 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1483) took 2222 ms. Trace:
I20260806 20:35:19.213073 16206 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:19.235843 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 198) took 2508 ms. Trace:
I20260806 20:35:19.237730 14552 rpcz_store.cc:276] 0806 20:35:16.720086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:16.720236 (+ 150us) service_pool.cc:227] Handling call
0806 20:35:16.803809 (+ 83573us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:16.804253 (+ 444us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:17.018407 (+214154us) log.cc:842] Serialized 14 byte log entry
0806 20:35:18.345151 (+1326744us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:18.345707 (+ 556us) log.cc:842] Serialized 26 byte log entry
0806 20:35:18.345831 (+ 124us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:18.378309 (+ 32478us) log.cc:842] Serialized 14 byte log entry
0806 20:35:18.396843 (+ 18534us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:18.396929 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:18.655271 (+258342us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b1800264580. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05424ca 00007fb9b9c23f99 00007fb9b739bbcf 00007fb9b05cb7b6 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d
0806 20:35:19.228375 (+573104us) raft_consensus.cc:1671] finished
0806 20:35:19.228409 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.228577 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87648,"prepare.queue_time_us":343071,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":299728,"raft.run_cpu_time_us":504,"raft.run_wall_time_us":55574,"spinlock_wait_cycles":7197952,"thread_start_us":1337291,"threads_started":2,"wal-append.queue_time_us":1862081}
W20260806 20:35:19.231302 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 1.01823e+06us
I20260806 20:35:19.206626 16285 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.233724 14477 rpcz_store.cc:276] 0806 20:35:17.010874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.011380 (+ 506us) service_pool.cc:227] Handling call
0806 20:35:17.011683 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:17.011874 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:18.136249 (+1124375us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.232532 (+1096283us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:19.232606 (+ 74us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:19.232666 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.232923 (+ 257us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.233077 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":295535,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"thread_start_us":2191292,"threads_started":2}
I20260806 20:35:19.227218 14549 rpcz_store.cc:276] 0806 20:35:17.970969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.971077 (+ 108us) service_pool.cc:227] Handling call
0806 20:35:17.971335 (+ 258us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:17.971505 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:17.971743 (+ 238us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.210434 (+1238691us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:19.210506 (+ 72us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:19.210565 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.210663 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.210830 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10073,"prepare.queue_time_us":3564,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":23820,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8715,"thread_start_us":1214620,"threads_started":1}
I20260806 20:35:19.289733 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:19.290526 14552 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:19.238539 16184 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:19.229894 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:19.232854 16280 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:19.293977 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:19.281553 15179 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.051s user 0.011s sys 0.056s
W20260806 20:35:19.270066 15946 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.029s user 0.018s sys 0.068s
W20260806 20:35:19.338798 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.006s user 0.024s sys 0.050s
I20260806 20:35:19.364478 16262 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:19.365134 16262 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.367313 16262 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:19.368914 16307 log.cc:924] Time spent T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20260806 20:35:19.370267 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:19.370954 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 425) took 2531 ms. Trace:
I20260806 20:35:19.311475 14549 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
W20260806 20:35:19.371917 16316 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260806 20:35:19.372439 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:19.372869 16302 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260806 20:35:19.373162 14549 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:19.374058 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:19.374347 16312 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20260806 20:35:19.374827 14549 raft_consensus.cc:2396] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:19.376787 16295 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:19.340550 15946 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.100s user 0.018s sys 0.068s
W20260806 20:35:19.363662 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.031s user 0.024s sys 0.053s
I20260806 20:35:19.381227 16295 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.381734 16295 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.375650 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:19.384299 14549 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:19.386596 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.387217 14633 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
W20260806 20:35:19.338406 15179 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.108s user 0.011s sys 0.057s
I20260806 20:35:19.389449 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:19.390053 14552 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:19.391489 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.392503 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.371461 15186 rpcz_store.cc:276] 0806 20:35:16.839731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:16.839841 (+ 110us) service_pool.cc:227] Handling call
0806 20:35:16.876198 (+ 36357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:16.958744 (+ 82546us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:16.958780 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:17.332516 (+373736us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x7b2c000f7568. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:17.337772 (+ 5256us) log.cc:842] Serialized 26 byte log entry
0806 20:35:18.514302 (+1176530us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:18.514357 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:18.514446 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.370578 (+856132us) raft_consensus.cc:1671] finished
0806 20:35:19.370609 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.370863 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82173,"raft.queue_time_us":90924,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":45981,"spinlock_wait_cycles":18343808,"thread_start_us":1176083,"threads_started":1,"wal-append.queue_time_us":1789422}
I20260806 20:35:19.417330 16261 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.131654 14546 rpcz_store.cc:276] 0806 20:35:17.087019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.145739 (+ 58720us) service_pool.cc:227] Handling call
0806 20:35:17.264391 (+118652us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:17.351620 (+ 87229us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:17.352032 (+ 412us) log.cc:842] Serialized 14 byte log entry
0806 20:35:18.777520 (+1425488us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:18.785047 (+ 7527us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:18.794166 (+ 9119us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.080134 (+285968us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b18002ef8a0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b0598f96 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9
0806 20:35:19.084666 (+ 4532us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b54002738d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:19.084764 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.109020 (+ 24256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4644,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":12039,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":11778,"spinlock_wait_cycles":168887808,"thread_start_us":1412605,"threads_started":1}
W20260806 20:35:19.449292 14478 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.186s user 0.025s sys 0.036s
I20260806 20:35:19.338660 16184 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:19.549578 14521 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.009s user 0.002s sys 0.029s
W20260806 20:35:19.550122 14598 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.012s sys 0.023s
W20260806 20:35:19.562913 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.118s user 0.001s sys 0.017s
I20260806 20:35:19.358229 15821 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260806 20:35:19.357693 15183 raft_consensus.cc:2362] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3. Re-sending same reply.
I20260806 20:35:19.583725 14629 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260806 20:35:19.339056 14547 raft_consensus.cc:2396] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:19.585114 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:19.519634 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.713s user 0.008s sys 0.012s
W20260806 20:35:19.641345 14521 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.101s user 0.003s sys 0.034s
I20260806 20:35:19.594362 14632 raft_consensus.cc:2379] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:19.604135 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:19.641718 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.835s user 0.009s sys 0.017s
I20260806 20:35:19.593964 14549 raft_consensus.cc:2379] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:19.642048 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 836064us
W20260806 20:35:19.642395 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.206s user 0.001s sys 0.018s
I20260806 20:35:19.642884 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:19.643369 14598 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.105s user 0.012s sys 0.029s
I20260806 20:35:19.643858 14629 pending_rounds.cc:76] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 4
I20260806 20:35:19.599680 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.644369 14629 pending_rounds.cc:97] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260806 20:35:19.645488 14633 raft_consensus.cc:2379] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:19.644835 14629 raft_consensus.cc:2866] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:35:19.647603 14478 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.385s user 0.025s sys 0.045s
W20260806 20:35:19.563514 15182 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.000s sys 0.017s
I20260806 20:35:19.668259 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1484) took 2629 ms. Trace:
W20260806 20:35:19.643271 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 1.16029e+06us
I20260806 20:35:19.668236 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:19.650414 14553 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.005s sys 0.007s
W20260806 20:35:19.569095 16187 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.007s sys 0.028s
W20260806 20:35:19.623848 16308 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.224s user 0.005s sys 0.009s
W20260806 20:35:19.643203 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 1.20817e+06us
W20260806 20:35:19.693461 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.010s sys 0.007s
W20260806 20:35:19.645162 16151 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.918s user 0.011s sys 0.019s
W20260806 20:35:19.696663 14548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.008s sys 0.051s
I20260806 20:35:19.699282 15151 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
W20260806 20:35:19.699867 14553 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.621s user 0.005s sys 0.013s
I20260806 20:35:19.584780 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.728941 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:19.648089 16167 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.019s sys 0.011s
W20260806 20:35:19.700750 15182 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.985s user 0.000s sys 0.019s
I20260806 20:35:19.736318 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.678087 16122 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:19.706949 16273 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:19.697194 14478 rpcz_store.cc:276] 0806 20:35:17.038772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.498282 (+459510us) service_pool.cc:227] Handling call
0806 20:35:17.498562 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:17.498755 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:17.605774 (+107019us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.667908 (+2062134us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:19.667968 (+ 60us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:19.668012 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.668080 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.668199 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":128003,"prepare.queue_time_us":119458,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":5478,"raft.queue_time_us":100773,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7056,"thread_start_us":1984948,"threads_started":2,"wal-append.queue_time_us":1045108}
I20260806 20:35:19.707724 16285 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.679162 16275 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:19.710620 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.748852 14629 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:19.711629 16184 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:19.616905 16261 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.754413 16261 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.755522 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 209) took 1784 ms. Trace:
I20260806 20:35:19.756124 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 168) took 2160 ms. Trace:
I20260806 20:35:19.743952 16298 consensus_queue.cc:1035] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260806 20:35:19.756608 14470 rpcz_store.cc:276] 0806 20:35:17.595376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.628064 (+ 32688us) service_pool.cc:227] Handling call
0806 20:35:17.628406 (+ 342us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:17.701505 (+ 73099us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:17.701547 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:17.719690 (+ 18143us) log.cc:842] Serialized 26 byte log entry
0806 20:35:19.010300 (+1290610us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:19.010356 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.078399 (+ 68043us) spinlock_profiling.cc:243] Waited 47.9 ms on lock 0x7b54003f57d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:19.078446 (+ 47us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.755814 (+677368us) raft_consensus.cc:1671] finished
0806 20:35:19.755838 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.756063 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29792,"raft.queue_time_us":30807,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"spinlock_wait_cycles":110124288,"thread_start_us":1283032,"threads_started":1,"wal-append.queue_time_us":1977107}
I20260806 20:35:19.758591 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.759202 14470 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:19.759373 16298 consensus_queue.cc:1046] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:19.712378 15181 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 422) took 2880 ms. Trace:
I20260806 20:35:19.735343 16110 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:19.761026 16110 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:19.714498 16338 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.646019 16206 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:19.720559 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.723140 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:19.724431 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.011s sys 0.007s
I20260806 20:35:19.741633 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 169) took 2121 ms. Trace:
I20260806 20:35:19.703907 14549 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:19.725950 14548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s user 0.008s sys 0.051s
W20260806 20:35:19.670135 14474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.014s sys 0.022s
I20260806 20:35:19.621884 15178 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 325) took 1338 ms. Trace:
W20260806 20:35:19.705804 16187 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.073s user 0.010s sys 0.031s
I20260806 20:35:19.785796 14548 raft_consensus.cc:1226] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:19.786414 16187 raft_consensus.cc:3032] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:19.786753 16187 raft_consensus.cc:729] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:19.787865 14548 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:19.787540 16187 consensus_queue.cc:260] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.670614 16217 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:19.788552 14474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s user 0.015s sys 0.023s
W20260806 20:35:19.669186 16193 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.047s user 0.020s sys 0.020s
W20260806 20:35:19.734694 16167 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s user 0.019s sys 0.011s
W20260806 20:35:19.789314 16193 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.203s user 0.020s sys 0.020s
I20260806 20:35:19.739971 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:19.725754 16151 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.009s user 0.011s sys 0.019s
I20260806 20:35:19.743189 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.702312 16067 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:19.794274 16067 raft_consensus.cc:3032] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:19.716776 16304 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:19.701095 15953 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.746605 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.715824 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:19.804476 15953 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.804756 14546 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:19.716373 16334 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.749987 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.809615 14626 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:19.715425 16263 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.751021 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.700858 16296 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.718591 16330 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:19.763266 14554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.011s sys 0.014s
W20260806 20:35:19.823477 14554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.012s sys 0.014s
I20260806 20:35:19.767024 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.824647 14478 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:19.746488 16122 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:19.778061 14627 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:19.756045 14550 rpcz_store.cc:276] 0806 20:35:17.970648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.973658 (+ 3010us) service_pool.cc:227] Handling call
0806 20:35:17.987267 (+ 13609us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:17.996595 (+ 9328us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:17.996627 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:17.997051 (+ 424us) log.cc:842] Serialized 26 byte log entry
0806 20:35:19.136751 (+1139700us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:19.136807 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.136902 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.755274 (+618372us) raft_consensus.cc:1671] finished
0806 20:35:19.755301 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.755448 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2131,"raft.queue_time_us":115,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"thread_start_us":1082404,"threads_started":1,"wal-append.queue_time_us":1742745}
I20260806 20:35:19.736299 14471 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:19.831838 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.749886 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 395) took 1548 ms. Trace:
I20260806 20:35:19.761439 16110 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:19.751401 16285 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.763353 16295 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.801607 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.844266 14477 raft_consensus.cc:2379] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:19.803397 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:19.799953 14476 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:19.777452 16184 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.799220 16187 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:19.794979 16067 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:19.839852 14555 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.834s user 0.013s sys 0.023s
W20260806 20:35:19.852084 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 500956bd1a3846a3a7b16c51ddecbc44 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:35:19.852377 14555 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s user 0.013s sys 0.023s
I20260806 20:35:19.835604 14629 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:19.850890 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.781847 16206 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.815016 16261 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.833666 16296 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:19.760145 15181 rpcz_store.cc:276] 0806 20:35:16.832038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:16.838339 (+ 6301us) service_pool.cc:227] Handling call
0806 20:35:16.881026 (+ 42687us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:18.324639 (+1443613us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:18.327855 (+ 3216us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:18.386618 (+ 58763us) log.cc:842] Serialized 26 byte log entry
0806 20:35:19.111007 (+724389us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:19.116736 (+ 5729us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.121261 (+ 4525us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.712074 (+590813us) raft_consensus.cc:1671] finished
0806 20:35:19.712120 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.712303 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91176,"raft.queue_time_us":141987,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":52882,"thread_start_us":2025638,"threads_started":2,"wal-append.queue_time_us":1315183}
I20260806 20:35:19.785543 15178 rpcz_store.cc:276] 0806 20:35:18.262253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:18.263653 (+ 1400us) service_pool.cc:227] Handling call
0806 20:35:19.504227 (+1240574us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54004c27d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:19.600889 (+ 96662us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1195030784}
I20260806 20:35:19.779639 16338 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.869650 16263 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:19.805259 15953 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.880954 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:19.858517 16318 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260806 20:35:19.847844 16049 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:19.858186 16295 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:19.886917 16031 consensus_meta.cc:287] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.809s user 0.014s sys 0.011s
I20260806 20:35:19.805351 16304 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.886871 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:19.896334 16049 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.902509 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:19.847963 16285 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:19.904268 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 207355us
I20260806 20:35:19.832891 16334 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.906302 14479 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:19.907920 14626 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:19.913827 16354 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:19.914299 16354 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.848878 16122 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.916271 16354 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.917132 16122 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.781576 14475 rpcz_store.cc:276] 0806 20:35:17.617645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.640008 (+ 22363us) service_pool.cc:227] Handling call
0806 20:35:17.696865 (+ 56857us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:17.714615 (+ 17750us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:17.716879 (+ 2264us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:17.752484 (+ 35605us) log.cc:842] Serialized 26 byte log entry
0806 20:35:19.009444 (+1256960us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:19.022653 (+ 13209us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.044735 (+ 22082us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.722545 (+677810us) raft_consensus.cc:1671] finished
0806 20:35:19.727891 (+ 5346us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.738563 (+ 10672us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5637,"raft.queue_time_us":10273,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":6855,"thread_start_us":1192848,"threads_started":1,"wal-append.queue_time_us":1956495}
I20260806 20:35:19.917654 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.918386 14629 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:19.889078 16296 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.905756 16067 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.919934 16067 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:19.920621 16296 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.922744 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.925302 14629 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:19.926882 16275 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:19.927487 16275 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:19.898708 16292 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:19.929090 16292 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:19.939162 16217 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:19.939713 16217 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:19.840191 16330 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.941077 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:19.943738 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:19.908615 16295 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.945614 14548 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:19.858516 16184 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:19.949635 16049 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:19.950461 16184 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:19.951050 14470 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:19.953075 16357 consensus_queue.cc:1035] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:19.953639 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:19.954185 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1408) took 1391 ms. Trace:
I20260806 20:35:19.954636 14474 rpcz_store.cc:276] 0806 20:35:18.563221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:19.053817 (+490596us) service_pool.cc:227] Handling call
0806 20:35:19.054201 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:19.054648 (+ 447us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:19.054693 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:19.055245 (+ 552us) log.cc:842] Serialized 26 byte log entry
0806 20:35:19.788975 (+733730us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:19.789026 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.789115 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.953981 (+164866us) raft_consensus.cc:1671] finished
0806 20:35:19.954007 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.954130 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":29079,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":211,"thread_start_us":733296,"threads_started":1,"wal-append.queue_time_us":809541}
W20260806 20:35:19.957784 16342 log.cc:924] Time spent T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260806 20:35:19.955674 16334 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:19.954293 14633 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:19.969866 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 464429us
I20260806 20:35:19.970628 14442 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:19.971692 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:19.972324 14627 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:19.984555 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:19.985626 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1295) took 1039 ms. Trace:
I20260806 20:35:19.986613 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.987231 14630 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:19.987980 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.989095 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:19.989432 14470 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:19.990964 15153 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:19.993439 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:19.993872 14550 raft_consensus.cc:1226] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:19.993456 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:19.998759 14625 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:35:19.986223 14553 rpcz_store.cc:276] 0806 20:35:18.946606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:19.078274 (+131668us) service_pool.cc:227] Handling call
0806 20:35:19.078594 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:19.078794 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:19.726306 (+647512us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.985331 (+259025us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:19.985380 (+ 49us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:19.985419 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.985478 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.985576 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":638866,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":888867,"threads_started":2}
W20260806 20:35:19.996342 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 263302us
I20260806 20:35:20.005496 14470 pending_rounds.cc:76] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 4
I20260806 20:35:20.005872 14470 pending_rounds.cc:97] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 20:35:19.966274 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 424) took 3120 ms. Trace:
I20260806 20:35:19.895699 16263 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.007193 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.006244 14470 raft_consensus.cc:2866] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:19.981020 16280 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:20.010964 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 231899us
I20260806 20:35:19.968719 14473 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:20.020753 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.021488 14632 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:20.021929 14596 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.022022 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.022482 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.022797 14625 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:20.023265 14632 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:20.023844 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.024638 14625 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:19.838626 15179 rpcz_store.cc:276] 0806 20:35:18.190702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:18.190939 (+ 237us) service_pool.cc:227] Handling call
0806 20:35:18.200874 (+ 9935us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:18.229451 (+ 28577us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:18.229500 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:18.230228 (+ 728us) log.cc:842] Serialized 26 byte log entry
0806 20:35:19.633517 (+1403289us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:19.640815 (+ 7298us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.675316 (+ 34501us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.722870 (+ 47554us) raft_consensus.cc:1671] finished
0806 20:35:19.728194 (+ 5324us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.739106 (+ 10912us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1361047,"threads_started":1,"wal-append.queue_time_us":1475668}
I20260806 20:35:20.036476 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.038919 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.039834 14630 raft_consensus.cc:2379] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:20.024252 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.007354 16285 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.008594 16332 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:20.007053 16206 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:20.069029 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.071125 14630 raft_consensus.cc:2379] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 2: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:20.073509 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.075631 14630 raft_consensus.cc:2396] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:20.019122 16304 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:20.082798 14553 raft_consensus.cc:1226] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:35:20.086174 14596 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.089634 14625 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:20.092679 14632 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.097738 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 206) took 2128 ms. Trace:
I20260806 20:35:20.010543 16273 consensus_queue.cc:1046] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 20:35:20.103968 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.110796 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.111550 14630 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:20.102941 14596 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.014022 15821 consensus_queue.cc:1046] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260806 20:35:20.063556 15953 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.105808 14551 rpcz_store.cc:276] 0806 20:35:17.969735 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.971363 (+ 1628us) service_pool.cc:227] Handling call
0806 20:35:17.971603 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:17.971761 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:19.216070 (+1244309us) log.cc:842] Serialized 14 byte log entry
0806 20:35:20.096528 (+880458us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:20.096616 (+ 88us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:20.097395 (+ 779us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:20.097491 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:20.097660 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29408,"prepare.queue_time_us":17133,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":98281,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":2093805,"threads_started":3,"wal-append.queue_time_us":680534}
W20260806 20:35:20.113271 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 76901.1us
I20260806 20:35:20.113229 14596 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.063772 16225 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:20.119426 14630 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:20.080598 16298 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:20.137359 16298 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.137466 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.100103 16352 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:20.105082 16296 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:35:20.142041 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 232377us
I20260806 20:35:20.101238 16295 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.156019 14596 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:20.139937 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 73976.6us
I20260806 20:35:20.156677 14626 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
W20260806 20:35:20.157181 14595 connection.cc:232] RPC call timeout handler was delayed by -1.1484338399895933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20260806 20:35:20.157492 14595 connection.cc:232] RPC call timeout handler was delayed by -1.1376338400004897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
W20260806 20:35:20.157752 14595 connection.cc:232] RPC call timeout handler was delayed by -1.0359338400012348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
I20260806 20:35:20.147765 14627 raft_consensus.cc:2396] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:20.116720 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.118990 16303 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:20.136350 16304 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:20.169885 16304 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:20.170223 16340 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20260806 20:35:20.170401 16304 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.171509 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.174314 14631 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.177021 16338 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.177579 16295 consensus_queue.cc:1046] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:20.138775 16225 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:20.191509 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:20.192130 14551 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.192313 14444 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.163636 14595 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.117879 16292 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.135357 16330 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.182466 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:20.181540 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.198547 14548 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:20.165860 14627 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:20.166750 14596 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.167323 16356 consensus_queue.cc:1046] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:20.202255 14518 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:20.168761 16343 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260806 20:35:20.084175 16332 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:20.161017 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 163720us
I20260806 20:35:20.178473 16295 raft_consensus.cc:3032] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:20.222239 16295 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:20.222142 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.180962 16296 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:20.223260 16295 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.223946 16296 raft_consensus.cc:686] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:20.164315 16122 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:20.224628 16122 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:20.225061 16122 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:20.224892 16296 consensus_queue.cc:237] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.165767 16303 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:20.197115 14595 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.230952 16077 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:20.231513 16077 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:20.231902 16077 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.230964 16303 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:20.234467 14595 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.010231 15182 rpcz_store.cc:276] 0806 20:35:16.839380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:16.875163 (+ 35783us) service_pool.cc:227] Handling call
0806 20:35:16.875595 (+ 432us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:16.964254 (+ 88659us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:17.001863 (+ 37609us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:17.009171 (+ 7308us) log.cc:842] Serialized 52 byte log entry
0806 20:35:18.709009 (+1699838us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:19.739371 (+1030362us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.892032 (+152661us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.892270 (+ 238us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.898419 (+ 6149us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:19.903747 (+ 5328us) raft_consensus.cc:1671] finished
0806 20:35:19.903764 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:19.960207 (+ 56443us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65930,"prepare.queue_time_us":320290,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":2415,"raft.queue_time_us":100239,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":43399,"thread_start_us":2818578,"threads_started":3,"wal-append.queue_time_us":2162910}
I20260806 20:35:20.190779 14521 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:20.203447 14596 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.206305 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.208199 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.201922 16049 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.208829 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:20.185150 16355 log.cc:924] Time spent T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260806 20:35:20.193764 16225 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.210009 16280 consensus_queue.cc:1046] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:20.168922 16298 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:20.211621 14444 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:20.288775 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 77248.6us
I20260806 20:35:20.217754 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.220516 16217 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:20.221297 16359 log.cc:924] Time spent T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260806 20:35:20.154867 16352 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.197574 15953 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:20.180338 16309 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:20.228823 15153 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.229738 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.258845 14518 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.266880 15186 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:20.185595 16263 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:20.222311 16348 log.cc:924] Time spent T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
W20260806 20:35:20.268400 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 77706.1us
I20260806 20:35:20.315625 14521 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.217345 15180 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:35:20.210976 16329 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:20.276203 14548 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:20.278781 14551 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:20.281455 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.201299 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.290022 14444 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.294561 14596 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.268720 16295 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:20.267616 16334 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:20.267313 16356 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:20.324728 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.266172 16304 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.328619 16262 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:20.260053 16360 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.356973 16304 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.299837 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.358301 14628 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:20.358981 14631 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:20.359570 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.359629 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.359987 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.291772 16186 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:20.312757 16292 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.366837 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.379669 16352 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:20.313462 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 54709.9us
I20260806 20:35:20.380810 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.381433 14518 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.371735 16330 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.322885 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.384109 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.385191 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:20.385462 16187 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.529s user 0.005s sys 0.006s
I20260806 20:35:20.386272 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:20.372290 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.387460 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.369544 16217 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:20.323318 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.330499 14521 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.368427 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:20.391013 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 60596.8us
I20260806 20:35:20.382565 14518 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:20.391978 14521 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.324297 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.336885 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.337822 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.338883 16122 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.315004 15953 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.331998 16049 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.340621 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.320324 16354 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.350757 16332 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:20.321547 16363 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:20.354246 16261 raft_consensus.cc:2741] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:20.354866 16303 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.453867 14627 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:20.272804 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1300) took 1262 ms. Trace:
I20260806 20:35:20.357641 15946 raft_consensus.cc:3032] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:20.360370 15179 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:20.438472 16067 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:20.347638 16110 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.332160 16334 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:20.364219 14628 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:20.364856 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 207) took 2394 ms. Trace:
W20260806 20:35:20.304378 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 75644.9us
I20260806 20:35:20.465968 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.366160 16344 raft_consensus.cc:2741] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:20.291930 16285 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260806 20:35:20.433589 15185 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:20.387279 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.329833 14444 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:20.471298 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 141548us
I20260806 20:35:20.472028 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.393448 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.393896 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:20.473497 14441 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.344182 14633 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:20.465505 14554 rpcz_store.cc:276] 0806 20:35:17.970042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:17.972168 (+ 2126us) service_pool.cc:227] Handling call
0806 20:35:17.972554 (+ 386us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:17.973007 (+ 453us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:19.138826 (+1165819us) log.cc:842] Serialized 14 byte log entry
0806 20:35:19.823890 (+685064us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:19.847049 (+ 23159us) log.cc:842] Serialized 26 byte log entry
0806 20:35:19.847216 (+ 167us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:19.847267 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:19.847372 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:20.364583 (+517211us) raft_consensus.cc:1671] finished
0806 20:35:20.364606 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:20.364793 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14741,"prepare.queue_time_us":16127,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":90339,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":17658,"spinlock_wait_cycles":854272,"thread_start_us":1796990,"threads_started":2,"wal-append.queue_time_us":829942}
I20260806 20:35:20.474541 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.366422 16275 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.434798 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:20.435400 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 52908.9us
I20260806 20:35:20.373099 16206 raft_consensus.cc:3032] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:20.372670 16179 raft_consensus.cc:3032] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:20.405575 14550 raft_consensus.cc:2379] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:20.430250 14632 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:20.442827 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 114839us
I20260806 20:35:20.339922 16077 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.382565 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.336282 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.442673 16298 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.306699 16309 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.466550 15153 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:20.454052 16352 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:20.455406 16049 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:20.535292 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.536187 14627 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:20.468603 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.331462 16356 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:20.536113 14476 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:20.475399 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.434594 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.467995 15178 raft_consensus.cc:2379] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:20.540580 14552 raft_consensus.cc:2379] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:20.476764 14625 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:20.439111 16280 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260806 20:35:20.545261 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.545970 14478 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:20.457348 15946 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:20.483801 14630 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:35:20.552536 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.486152 14444 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.552759 14630 raft_consensus.cc:595] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.553355 14630 raft_consensus.cc:603] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:20.512274 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:20.487841 15953 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.491472 16186 consensus_queue.cc:1046] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260806 20:35:20.485807 16179 raft_consensus.cc:729] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:20.490432 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.488279 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.490628 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.467830 16334 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:20.518554 14555 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:20.487198 14595 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.482064 16167 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:20.528966 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 62509.4us
I20260806 20:35:20.484977 16262 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.527817 14632 raft_consensus.cc:1261] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260806 20:35:20.605509 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 118406us
I20260806 20:35:20.605755 14632 raft_consensus.cc:1226] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:20.534873 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.606194 14595 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.606407 14477 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:20.538807 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.539477 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:20.536923 14628 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:20.542872 14470 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:20.453531 16364 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:20.470573 16217 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260806 20:35:20.518126 16338 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:20.491812 16151 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:20.553725 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 67660.1us
I20260806 20:35:20.553514 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.558086 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.493274 16351 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.602402 14474 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.557617 16309 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:20.554461 14630 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
W20260806 20:35:20.605918 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 214000us
I20260806 20:35:20.681365 14521 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.537894 16287 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.594558 14554 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:20.617340 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:20.683689 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 66444us
I20260806 20:35:20.617781 14518 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
W20260806 20:35:20.684626 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 66921.7us
I20260806 20:35:20.686120 14518 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:20.620397 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.687762 14555 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:20.554147 16077 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:20.623620 14632 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:20.625011 15153 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:20.554989 16383 log.cc:924] Time spent T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260806 20:35:20.489837 16225 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.484421 16206 raft_consensus.cc:729] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:20.629119 14470 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:20.622542 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:20.547663 16356 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:20.633342 14596 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.554127 16184 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:20.548017 16352 raft_consensus.cc:686] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:20.552707 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.554628 16110 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:20.653364 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.602866 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.561239 16122 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:20.555780 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.555341 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:20.592947 16363 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:20.654388 14444 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.653393 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:20.655360 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:20.725277 16344 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:20.625721 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.640925 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:20.641885 16151 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:20.687137 14518 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:20.719326 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:20.771771 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:20.773840 14477 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.770049 14479 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:35:20.775629 16151 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.735699 14595 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:20.735942 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 102677us
I20260806 20:35:20.670502 16362 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:20.725884 16390 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:20.736641 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 55399.5us
I20260806 20:35:20.616569 16296 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:20.673179 16273 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:20.725445 16357 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:20.724833 16298 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:20.745865 16358 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 20:35:20.779006 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 91950.1us
I20260806 20:35:20.785125 14521 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.740968 16303 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:20.789402 14518 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:20.743595 16184 raft_consensus.cc:686] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:20.789947 16381 log.cc:924] Time spent T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260806 20:35:20.751865 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:20.776304 16387 log.cc:924] Time spent T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
I20260806 20:35:20.769686 16391 raft_consensus.cc:2741] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:20.786145 16067 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 20:35:20.796571 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 126470us
I20260806 20:35:20.797868 14596 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.791147 16358 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:20.783341 15183 raft_consensus.cc:1226] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:20.726879 16179 consensus_queue.cc:260] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.790258 15182 raft_consensus.cc:2354] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3.
I20260806 20:35:20.783281 16362 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:20.806326 14477 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:20.796047 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 60447.7us
I20260806 20:35:20.668779 15946 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.791846 16157 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:20.786840 16390 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:20.785863 16285 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:20.783931 14473 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:20.792634 16357 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:20.790062 16280 raft_consensus.cc:3032] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:20.800258 16077 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:20.829231 16362 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.799697 16298 raft_consensus.cc:686] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:20.824018 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 199120us
I20260806 20:35:20.456220 14547 rpcz_store.cc:276] 0806 20:35:19.008489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:19.709948 (+701459us) service_pool.cc:227] Handling call
0806 20:35:19.710268 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:19.710476 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:20.003757 (+293281us) log.cc:842] Serialized 14 byte log entry
0806 20:35:20.193865 (+190108us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:20.201649 (+ 7784us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:20.212063 (+ 10414us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:20.230351 (+ 18288us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:20.270739 (+ 40388us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":7409,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":95,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":416468,"threads_started":2,"wal-append.queue_time_us":181579}
I20260806 20:35:20.828004 14595 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.855662 14595 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.856618 15953 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.857359 15953 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:20.857636 14595 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:20.857738 15953 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:20.810284 16067 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:20.858163 15953 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.858433 16067 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:20.795521 16273 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.845713 16357 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.817772 16358 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:20.799381 16275 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:20.849954 15946 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:20.844969 16157 raft_consensus.cc:686] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:20.847249 16077 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:20.830185 16384 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:20.840081 16382 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
W20260806 20:35:20.852710 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 67696.1us
I20260806 20:35:20.794441 16206 consensus_queue.cc:260] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.861323 14595 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.640089 16364 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.834743 16285 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:20.834614 16390 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:20.874192 16298 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.865828 16225 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:20.865592 16179 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:20.862504 16184 consensus_queue.cc:237] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.870808 16077 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:20.863242 16354 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.870666 16384 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:20.887568 14595 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.887638 14474 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:20.890375 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.863412 15153 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:20.851464 16295 consensus_meta.cc:287] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.516s user 0.014s sys 0.004s
W20260806 20:35:20.864749 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 75423.6us
I20260806 20:35:20.809018 16352 consensus_queue.cc:237] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.846238 16280 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:20.847153 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 206259us
I20260806 20:35:20.819990 16389 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:35:20.873622 16388 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20260806 20:35:20.905026 16280 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.873320 16157 consensus_queue.cc:237] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.906344 16280 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.874516 16285 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:20.892910 14521 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:20.895511 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 151573us
I20260806 20:35:20.889076 16384 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.896540 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.916035 14477 raft_consensus.cc:2379] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:20.770532 16363 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:20.898705 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.899674 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.900631 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.890758 16391 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:20.901852 15153 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:20.896822 16225 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:20.904438 14595 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.896306 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.888166 16262 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:20.920745 14474 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:20.921041 14479 raft_consensus.cc:2396] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:20.902081 14631 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:20.921784 15150 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:20.921918 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.936784 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.892916 16206 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:20.925401 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.923377 16391 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:20.908824 16360 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:20.922597 16049 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:20.934361 14595 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:20.932744 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.932854 16286 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:20.934899 16390 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:20.928684 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.902756 16354 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:20.908877 16389 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:20.944764 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.945986 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.931829 14475 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:20.958402 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:21.004366 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:20.956758 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.963449 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.948501 16391 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:20.935127 16357 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:20.923823 16292 consensus_meta.cc:287] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.547s user 0.017s sys 0.001s
I20260806 20:35:21.035074 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:20.967324 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:20.932420 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.038425 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:20.931594 16364 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:20.992329 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:20.992691 14629 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:20.995226 14471 raft_consensus.cc:1226] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:21.000202 15153 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.001405 14444 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.006513 14477 raft_consensus.cc:2379] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:20.941690 16333 consensus_queue.cc:1046] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260806 20:35:20.967231 16225 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:20.963538 16360 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:21.035415 14549 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:20.969784 16049 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:20.967950 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:21.008929 16286 raft_consensus.cc:686] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:21.059832 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 135173us
W20260806 20:35:21.064170 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 62849.2us
I20260806 20:35:20.943264 16262 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:21.073313 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 145105us
I20260806 20:35:21.041990 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:21.069320 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 69199.5us
I20260806 20:35:21.069847 14470 raft_consensus.cc:2379] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:21.074779 14474 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:20.948175 16397 consensus_queue.cc:1046] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260806 20:35:21.013196 16167 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:21.071187 14477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c"
dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47208
I20260806 20:35:21.074026 14444 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.035221 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:21.074939 14518 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:20.964597 16399 consensus_queue.cc:1035] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.139s
I20260806 20:35:20.948365 16403 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.144s
I20260806 20:35:21.003942 16389 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:21.062405 16292 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:21.064668 16393 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.075s user 0.000s sys 0.004s
I20260806 20:35:21.080493 14521 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.004099 16184 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:21.070107 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 135820us
I20260806 20:35:21.081670 14477 raft_consensus.cc:595] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:21.063833 16157 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:21.090648 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.090829 14470 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:35:21.071570 16049 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:21.009891 16352 consensus_queue.cc:1046] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260806 20:35:21.099742 14521 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.094175 14477 raft_consensus.cc:603] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:21.094328 14595 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:21.074100 16225 raft_consensus.cc:686] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:21.083015 16217 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:21.079136 16396 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.096630 14444 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.070878 16360 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:21.095657 16333 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.117044 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1257) took 1104 ms. Trace:
I20260806 20:35:21.106638 14629 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:21.093222 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.098742 16157 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:21.090332 16273 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.117545 14627 rpcz_store.cc:276] 0806 20:35:20.012320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.543684 (+531364us) service_pool.cc:227] Handling call
0806 20:35:20.544081 (+ 397us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:20.550539 (+ 6458us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:20.551272 (+ 733us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:20.551734 (+ 462us) log.cc:842] Serialized 26 byte log entry
0806 20:35:20.736371 (+184637us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:20.778071 (+ 41700us) log.cc:842] Serialized 14 byte log entry
0806 20:35:20.778304 (+ 233us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:20.778462 (+ 158us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:20.875919 (+ 97457us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b1800114820. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:21.109235 (+233316us) raft_consensus.cc:1671] finished
0806 20:35:21.116784 (+ 7549us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:21.116974 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":41455,"prepare.run_wall_time_us":42,"raft.queue_time_us":11104,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":4321,"spinlock_wait_cycles":11026176,"thread_start_us":184317,"threads_started":1,"wal-append.queue_time_us":234765}
I20260806 20:35:21.142869 14474 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:21.085575 16167 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.098234 16184 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.185791 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.109772 16412 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:21.342165 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 101607us
W20260806 20:35:21.258838 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 159159us
I20260806 20:35:21.345023 14521 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.170753 15153 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:21.149644 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 56502.7us
I20260806 20:35:21.349968 14518 leader_election.cc:398] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3.
I20260806 20:35:21.228869 16357 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.355536 16357 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:21.202593 16406 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20260806 20:35:21.201766 16407 log.cc:924] Time spent T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260806 20:35:21.205399 16261 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:21.168406 16402 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:21.148017 16333 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:21.364630 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 193986us
W20260806 20:35:21.345248 16394 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.369s user 0.004s sys 0.000s
I20260806 20:35:21.350589 14518 leader_election.cc:401] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:21.113752 16408 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
W20260806 20:35:21.377573 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 283301us
I20260806 20:35:21.175410 16411 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.190857 16416 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:21.152757 16410 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:21.175287 16363 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:21.385952 14479 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 1.
I20260806 20:35:21.215790 15953 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.203205 16077 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.112829 16217 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.380908 16361 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 20:35:21.394618 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 44825us
I20260806 20:35:21.213873 16067 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.434244 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:21.344591 16358 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.380707 16402 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:21.380195 16261 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:21.407181 16410 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:21.413981 16398 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:21.414513 16184 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.400430 16403 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:35:21.112003 16356 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.423369 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:21.423560 16409 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:21.348220 16384 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.425004 15181 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:21.425879 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:21.426766 16361 raft_consensus.cc:686] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:21.426923 16287 consensus_meta.cc:287] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.701s user 0.011s sys 0.000s
I20260806 20:35:21.427879 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:21.468845 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:21.407884 16408 raft_consensus.cc:686] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:21.431152 16285 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.363965 16341 consensus_queue.cc:1046] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 20:35:21.432879 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:21.433161 16077 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.191689 16390 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.120569 16286 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.186791 16362 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.452065 14552 raft_consensus.cc:1226] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:21.453876 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:21.454169 14518 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.188272 16225 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.488896 16285 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:21.490875 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:21.441983 16261 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:21.445106 16402 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:21.444155 16410 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:21.419061 16049 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.458187 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.486075 14477 raft_consensus.cc:1261] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:21.488312 16332 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:21.495632 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 150706us
I20260806 20:35:21.472209 16287 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.491947 16408 consensus_queue.cc:237] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.498467 14473 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:21.520299 16151 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.744s user 0.005s sys 0.007s
I20260806 20:35:21.545143 14521 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.452211 16398 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.458560 16409 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.551853 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:21.548553 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.525166 16273 consensus_queue.cc:1046] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260806 20:35:21.549641 15953 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:21.549875 16217 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.529430 16332 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.552542 14518 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.556675 14627 raft_consensus.cc:2354] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3.
W20260806 20:35:21.551852 16334 consensus_meta.cc:287] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.921s user 0.008s sys 0.009s
W20260806 20:35:21.596987 16418 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260806 20:35:21.527490 14624 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:21.553197 16206 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.610s user 0.010s sys 0.001s
W20260806 20:35:21.599149 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 46716.3us
I20260806 20:35:21.601071 14444 leader_election.cc:398] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3.
I20260806 20:35:21.551012 16167 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.498077 16390 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:21.564158 16415 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.611831 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.646836 14628 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:21.648856 15153 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:21.634279 14444 leader_election.cc:401] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260806 20:35:21.607651 16389 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.509s user 0.005s sys 0.005s
I20260806 20:35:21.639191 14630 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:35:21.656940 14630 raft_consensus.cc:595] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.657418 14630 raft_consensus.cc:603] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:21.660128 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:21.641201 14442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.029s sys 0.065s
I20260806 20:35:21.602314 16416 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:21.612543 16354 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.003s sys 0.027s
I20260806 20:35:21.560022 16352 consensus_queue.cc:1035] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260806 20:35:21.547312 16151 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:21.650449 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 82224.1us
W20260806 20:35:21.662192 14442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.662s user 0.029s sys 0.066s
I20260806 20:35:21.673300 14444 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:21.673422 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 673089us
I20260806 20:35:21.674019 14624 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:21.674878 16184 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:21.675275 16184 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:21.675534 14630 raft_consensus.cc:1261] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:21.675644 16184 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:21.663513 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 61106.1us
I20260806 20:35:21.664585 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.676777 14632 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:21.677896 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:21.679680 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.682861 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:21.685189 16420 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260806 20:35:21.686551 15178 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:21.687897 16332 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.689023 16151 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:21.689800 16049 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.634661 16363 consensus_queue.cc:1046] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
W20260806 20:35:21.666704 16354 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.003s sys 0.027s
I20260806 20:35:21.694113 14548 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:21.722054 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:21.694633 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:21.723080 14548 raft_consensus.cc:2379] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:21.728946 14553 raft_consensus.cc:1261] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:21.624883 16334 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.703096 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.704178 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.734126 14627 raft_consensus.cc:2379] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:21.734202 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.735175 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.736171 14627 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:21.706148 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:21.666348 16415 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:21.659840 16273 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:35:21.664970 16292 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:21.747720 16292 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:21.748165 16292 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:21.708783 15946 consensus_meta.cc:287] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.828s user 0.010s sys 0.002s
I20260806 20:35:21.711370 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:21.712469 16360 consensus_meta.cc:287] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.592s user 0.010s sys 0.000s
I20260806 20:35:21.754076 15185 raft_consensus.cc:2354] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
I20260806 20:35:21.754302 16360 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.756414 14476 raft_consensus.cc:2396] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:35:21.660980 16389 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.712862 14442 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.715384 15151 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.715884 16184 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.717202 16167 consensus_queue.cc:1046] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:21.717664 16384 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:21.718041 16309 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:21.718690 16398 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.657291 16067 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.650651 16358 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.553504 16361 consensus_queue.cc:237] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.700420 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:21.733304 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 54716.7us
I20260806 20:35:21.784087 14441 leader_election.cc:398] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
I20260806 20:35:21.784574 14441 leader_election.cc:401] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 20:35:21.737641 14626 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:21.739073 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.727185 16354 raft_consensus.cc:686] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:21.741653 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.691448 14632 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:21.753204 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.760263 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:21.760437 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:21.763260 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 50487.4us
I20260806 20:35:21.749300 16415 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.791292 14553 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:21.767618 16429 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.765661 16263 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:21.774109 16410 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.745216 16362 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:21.781301 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.791139 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.819271 14627 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:35:21.822570 14625 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:21.822660 14627 raft_consensus.cc:595] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:21.823155 14627 raft_consensus.cc:603] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:21.791146 14633 raft_consensus.cc:2354] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
I20260806 20:35:21.785593 15150 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.768966 16409 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:21.820791 14473 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:21.825003 14444 leader_election.cc:398] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
W20260806 20:35:21.832190 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 46707.4us
I20260806 20:35:21.835994 14625 raft_consensus.cc:1261] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:21.839584 14442 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:21.840523 16292 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.842137 16179 raft_consensus.cc:3032] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:21.842280 16403 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3.
I20260806 20:35:21.842586 16179 raft_consensus.cc:729] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:21.842772 16184 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.843547 16179 consensus_queue.cc:260] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.846817 16186 raft_consensus.cc:3032] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:21.847172 16186 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:21.801720 16309 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:21.847944 16186 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:21.849835 16332 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:21.819569 16402 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.853389 16402 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.853506 16151 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:21.854400 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1520) took 1315 ms. Trace:
I20260806 20:35:21.854441 16151 raft_consensus.cc:3032] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:21.855019 16151 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:21.854921 14476 rpcz_store.cc:276] 0806 20:35:20.538783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.539162 (+ 379us) service_pool.cc:227] Handling call
0806 20:35:21.854191 (+1315029us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001a59d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:21.854320 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2641562368}
I20260806 20:35:21.855659 16415 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:21.819064 14547 raft_consensus.cc:1226] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:21.856536 16263 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.857810 14555 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:21.858528 14554 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:21.851168 16434 consensus_queue.cc:1046] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:21.863111 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:21.864888 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:21.866861 14627 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:21.861842 14518 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.869020 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:21.868741 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 128303us
I20260806 20:35:21.870011 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:21.818202 16356 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:21.873889 14627 raft_consensus.cc:1261] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:21.865998 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 166352us
I20260806 20:35:21.855705 16410 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:21.865893 16391 consensus_meta.cc:287] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.826s user 0.008s sys 0.004s
I20260806 20:35:21.889292 15153 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:21.885695 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:21.889945 14625 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:21.854333 16334 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:21.848804 16360 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:21.868697 16167 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:21.894366 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:21.899400 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:21.900816 14521 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.855918 16151 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.902525 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:21.905684 14473 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:21.875535 16415 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:21.840445 16280 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260806 20:35:21.912497 14521 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.922864 14521 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.925033 14521 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.927881 14625 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:21.929322 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:21.930790 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:21.933225 16151 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:21.908186 14632 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:21.937377 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:21.938562 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:21.938594 16440 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:21.938768 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 451) took 1023 ms. Trace:
I20260806 20:35:21.891834 16391 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.940557 14628 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:21.939429 15179 rpcz_store.cc:276] 0806 20:35:20.914913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.915496 (+ 583us) service_pool.cc:227] Handling call
0806 20:35:21.938046 (+1022550us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:21.938252 (+ 206us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.938426 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:21.938457 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:21.938509 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:21.938541 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:21.938604 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:21.938717 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10772864}
I20260806 20:35:21.941931 16391 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:21.944841 14548 raft_consensus.cc:3032] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:21.945379 14548 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:21.946271 14548 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.947641 14548 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:21.949540 16389 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:21.949795 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1527) took 1314 ms. Trace:
I20260806 20:35:21.950058 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:21.950192 14471 rpcz_store.cc:276] 0806 20:35:20.635588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.635706 (+ 118us) service_pool.cc:227] Handling call
0806 20:35:20.636008 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.064510 (+428502us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:21.949594 (+885084us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b5400332cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:21.949741 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":825704704}
I20260806 20:35:21.918205 16402 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:21.952346 16410 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.953225 16356 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.954057 16067 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:21.950680 14473 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:21.932191 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:21.965220 14518 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.966214 14444 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:21.948132 16273 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3.
I20260806 20:35:21.971696 15153 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:21.853415 16354 consensus_queue.cc:237] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:21.971758 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.904994 16167 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:21.976548 14521 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.992079 14521 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:21.986446 16440 raft_consensus.cc:686] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:22.000408 16389 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:22.010560 16286 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:22.010823 16273 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:22.011119 16286 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:22.011610 16286 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:22.012898 16415 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:22.012938 14521 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:22.025620 14521 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:22.029353 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:22.029842 15151 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.973609 16067 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:22.030707 14551 raft_consensus.cc:2379] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:22.031093 15151 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:21.997488 16167 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:21.991595 16402 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:21.983608 16410 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:22.071312 16408 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:22.074034 14474 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:22.079403 14548 raft_consensus.cc:1261] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:22.031562 16352 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:22.096623 16424 consensus_queue.cc:1046] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:35:22.089713 16309 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4.
I20260806 20:35:22.091277 16217 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:21.955895 15946 consensus_queue.cc:1159] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20260806 20:35:22.091796 16447 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:22.095009 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:22.099697 14471 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:22.031324 16437 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:22.071909 16408 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.063160 16389 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:35:22.112943 14473 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:22.097194 16401 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 1.239s user 0.002s sys 0.001s
W20260806 20:35:22.082782 16400 log.cc:924] Time spent T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.223s user 0.003s sys 0.000s
W20260806 20:35:22.100492 16261 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.608s user 0.002s sys 0.010s
I20260806 20:35:22.138310 15185 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 20:35:22.092424 16448 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:22.132822 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:22.018656 16440 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.091984 16167 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:22.109993 16403 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:22.141662 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 244409us
I20260806 20:35:22.073719 16067 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:22.137382 16363 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260806 20:35:22.329121 14518 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:22.330477 16273 raft_consensus.cc:686] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:22.131371 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 143256us
I20260806 20:35:22.136422 16399 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:22.343488 14548 raft_consensus.cc:2362] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3. Re-sending same reply.
I20260806 20:35:22.363016 14477 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:22.315531 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 444) took 1844 ms. Trace:
W20260806 20:35:22.346046 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 314992us
I20260806 20:35:22.131521 16356 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:22.337752 16389 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:22.367844 16049 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:22.424216 16273 consensus_queue.cc:237] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:22.337924 16424 consensus_queue.cc:1046] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260806 20:35:22.434965 15151 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:22.347414 16448 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:22.430763 16445 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260806 20:35:22.458578 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1528) took 1822 ms. Trace:
I20260806 20:35:22.461295 14551 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:22.328056 16447 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:22.467101 16449 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20260806 20:35:22.441589 16423 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:22.480593 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 446) took 1843 ms. Trace:
I20260806 20:35:22.483232 15186 rpcz_store.cc:276] 0806 20:35:20.637648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.637778 (+ 130us) service_pool.cc:227] Handling call
0806 20:35:20.638092 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:20.665521 (+ 27429us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:20.665554 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:20.666179 (+ 625us) log.cc:842] Serialized 26 byte log entry
0806 20:35:20.862672 (+196493us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:20.862717 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:20.862785 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:22.096542 (+1233757us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:22.480225 (+383683us) raft_consensus.cc:1671] finished
0806 20:35:22.480253 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:22.480526 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27033,"raft.queue_time_us":27268,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":56404,"spinlock_wait_cycles":24565120,"thread_start_us":130906,"threads_started":1,"wal-append.queue_time_us":182900}
I20260806 20:35:22.381986 14548 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:22.339365 16261 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.361865 16403 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.428682 16356 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.498406 16356 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:22.498770 16403 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:22.430217 16389 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:22.503127 16389 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:22.458266 16309 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 20:35:22.506088 16408 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:22.505816 16286 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.509790 16423 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:22.510450 16261 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:22.513535 16167 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:22.367202 15183 rpcz_store.cc:276] 0806 20:35:20.298904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.301125 (+ 2221us) service_pool.cc:227] Handling call
0806 20:35:20.328006 (+ 26881us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:20.819602 (+491596us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:22.130519 (+1310917us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b540005fd54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:22.142798 (+ 12279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":962318720}
I20260806 20:35:22.515189 16262 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:22.503238 16451 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:22.518211 14369 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45942 (request call id 214) took 15489 ms (client timeout 15000 ms). Trace:
I20260806 20:35:22.480031 16437 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:22.475862 14478 rpcz_store.cc:276] 0806 20:35:20.636547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.674507 (+ 37960us) service_pool.cc:227] Handling call
0806 20:35:20.729678 (+ 55171us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:20.729978 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:20.730007 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:20.743040 (+ 13033us) log.cc:842] Serialized 26 byte log entry
0806 20:35:20.859488 (+116448us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:20.860175 (+ 687us) log.cc:842] Serialized 14 byte log entry
0806 20:35:20.860546 (+ 371us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:20.862217 (+ 1671us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:22.082232 (+1220015us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:22.443479 (+361247us) raft_consensus.cc:1671] finished
0806 20:35:22.458307 (+ 14828us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:22.458484 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8011,"prepare.queue_time_us":167,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":8799,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":9206,"spinlock_wait_cycles":24022272,"thread_start_us":64280,"threads_started":1,"wal-append.queue_time_us":88784}
W20260806 20:35:22.468330 16151 consensus_meta.cc:287] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.535s user 0.009s sys 0.000s
I20260806 20:35:22.499819 16403 consensus_queue.cc:1046] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:35:22.482699 16049 raft_consensus.cc:686] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:22.488545 16456 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:22.493561 16453 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 20:35:22.517668 16451 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.520920 16437 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.373625 16399 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:22.526501 16286 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:22.471662 14474 raft_consensus.cc:1261] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:22.470707 16361 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.003s sys 0.007s
I20260806 20:35:22.519729 16447 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:22.539239 16361 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.003s sys 0.007s
I20260806 20:35:22.526293 16049 consensus_queue.cc:237] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:22.518648 14369 rpcz_store.cc:269] 0806 20:35:07.028982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:07.029178 (+ 196us) service_pool.cc:227] Handling call
0806 20:35:22.518113 (+15488935us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:19.379587 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:19.380276 (+ 689us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:19.380327 (+ 51us) write_op.cc:435] Acquired schema lock
0806 20:35:19.380353 (+ 26us) tablet.cc:660] Decoding operations
0806 20:35:19.381103 (+ 750us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:19.381188 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:19.381228 (+ 40us) tablet.cc:683] Acquiring locks for 3 operations
0806 20:35:19.381659 (+ 431us) tablet.cc:699] Row locks acquired
0806 20:35:19.381687 (+ 28us) write_op.cc:260] PREPARE: finished
0806 20:35:19.381869 (+ 182us) write_op.cc:270] Start()
0806 20:35:19.382016 (+ 147us) write_op.cc:276] Timestamp: P: 1786048519381814 usec, L: 0
0806 20:35:19.382046 (+ 30us) op_driver.cc:331] REPLICATION: starting
0806 20:35:19.450610 (+ 68564us) log.cc:842] Serialized 1307 byte log entry
0806 20:35:20.462627 (+1012017us) op_driver.cc:444] REPLICATION: finished
0806 20:35:20.817334 (+354707us) write_op.cc:301] APPLY: starting
0806 20:35:20.819661 (+ 2327us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:20.834777 (+ 15116us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:20.842004 (+ 7227us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:20.842710 (+ 706us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:20.842944 (+ 234us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 20:35:20.843011 (+ 67us) write_op.cc:312] APPLY: finished
0806 20:35:20.843282 (+ 271us) log.cc:842] Serialized 35 byte log entry
0806 20:35:20.843461 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:20.843552 (+ 91us) write_op.cc:454] Released schema lock
0806 20:35:20.843598 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:20.843865 (+ 267us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:20.843892 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":850858,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":317212,"apply.run_cpu_time_us":2882,"apply.run_wall_time_us":39260,"num_ops":3,"prepare.queue_time_us":1754783,"prepare.run_cpu_time_us":38819,"prepare.run_wall_time_us":624345,"raft.queue_time_us":103812,"raft.run_cpu_time_us":34509,"raft.run_wall_time_us":264304,"replication_time_us":1075087,"thread_start_us":1646266,"threads_started":4,"wal-append.queue_time_us":485299,"wal-append.run_cpu_time_us":13364,"wal-append.run_wall_time_us":1788816}]]}
I20260806 20:35:22.579241 14474 consensus_queue.cc:237] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:22.583791 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 107071us
I20260806 20:35:22.590397 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:22.591266 14551 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:22.580593 14504 heartbeater.cc:644] Failed to heartbeat to 127.12.248.190:39439 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.12.248.190:39439 timed out after 15.000s (SENT)
I20260806 20:35:22.543633 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:22.599207 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 7f41753baced49a1997940dfc28b2e00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:22.541093 16397 consensus_queue.cc:1046] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 20:35:22.597738 14548 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:22.607291 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:22.605448 16460 consensus_queue.cc:1046] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.012s
I20260806 20:35:22.619135 14551 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:22.597324 16423 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:22.622478 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:22.599217 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:22.606474 16151 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:22.621770 16462 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:22.725407 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:22.728933 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 144220us
I20260806 20:35:22.725737 14548 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:22.728083 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 109311us
I20260806 20:35:22.725961 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:22.729166 14553 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:22.732591 16451 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:22.731081 14478 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:22.733793 16448 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:22.735077 16462 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:22.734125 16463 log.cc:924] Time spent T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.145s user 0.000s sys 0.007s
I20260806 20:35:22.765447 16151 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:22.912698 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 181760us
I20260806 20:35:22.781534 16397 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:22.780447 16217 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:22.730059 16460 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.180s
I20260806 20:35:22.847535 16464 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:22.768347 15151 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:22.937357 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 147786us
W20260806 20:35:22.949025 16405 log.cc:924] Time spent T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.852s user 0.000s sys 0.011s
I20260806 20:35:22.951908 16437 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:22.933350 16309 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:22.956647 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 188419us
I20260806 20:35:22.944721 16456 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:22.769614 16447 raft_consensus.cc:3032] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:22.969431 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:22.987670 14631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1590) took 2803 ms (client timeout 1683 ms). Trace:
I20260806 20:35:22.979036 16309 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:22.960894 16217 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:22.991624 16424 raft_consensus.cc:3032] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:22.993562 14478 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:22.997193 14548 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:22.952499 15946 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.012422 16473 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.012688 15946 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:22.949471 16399 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:22.961205 16440 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:22.996903 16447 raft_consensus.cc:729] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:23.018368 16295 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.085s user 0.007s sys 0.012s
W20260806 20:35:23.002368 14631 rpcz_store.cc:269] 0806 20:35:20.183812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.359745 (+175933us) service_pool.cc:227] Handling call
0806 20:35:22.987332 (+2627587us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b5400051254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:22.987550 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1193091456}
I20260806 20:35:23.038089 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1331) took 2123 ms. Trace:
I20260806 20:35:23.040994 14550 rpcz_store.cc:276] 0806 20:35:20.914632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.914744 (+ 112us) service_pool.cc:227] Handling call
0806 20:35:20.931900 (+ 17156us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:20.946427 (+ 14527us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:20.946472 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:20.946526 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:20.946578 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.037829 (+2091251us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b54000861d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:23.037889 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:23.038026 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32878336}
I20260806 20:35:23.041031 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:23.046257 16467 log.cc:924] Time spent T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260806 20:35:23.046175 15182 raft_consensus.cc:2354] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:23.013262 16473 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.050128 16462 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.050386 16309 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.051141 16473 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.008896 16424 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:23.048065 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 405) took 2135 ms. Trace:
I20260806 20:35:22.955449 16397 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:23.036041 16295 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.103s user 0.008s sys 0.013s
I20260806 20:35:23.053021 16424 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.073312 15946 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.053886 15182 rpcz_store.cc:276] 0806 20:35:20.913751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.916565 (+ 2814us) service_pool.cc:227] Handling call
0806 20:35:23.040872 (+2124307us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593bce 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:23.047885 (+ 7013us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12564864}
I20260806 20:35:22.997748 16464 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.135027 14632 raft_consensus.cc:3032] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:23.144055 14632 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:23.152674 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:23.153028 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:23.154090 15178 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:23.154702 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:23.155017 15151 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:23.155575 15182 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:23.156543 16423 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:23.157253 15187 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:35:23.080892 16462 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:23.086045 16465 consensus_queue.cc:1035] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.119s
I20260806 20:35:23.086966 16397 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.145453 14632 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.166308 16478 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.177709 14632 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:23.157146 16423 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.150955 16447 consensus_queue.cc:260] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.365141 16357 raft_consensus.cc:3032] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:23.161056 16292 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.368144 16357 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:23.357141 14549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1512) took 2735 ms (client timeout 3264 ms). Trace:
W20260806 20:35:23.368918 14549 rpcz_store.cc:269] 0806 20:35:20.622158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.622291 (+ 133us) service_pool.cc:227] Handling call
0806 20:35:23.356910 (+2734619us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b54000b40d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:23.357070 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":667769600}
I20260806 20:35:23.189884 14632 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:35:23.178551 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:23.359169 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1437) took 2367 ms (client timeout 2071 ms). Trace:
I20260806 20:35:23.359345 14441 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:23.370280 14479 rpcz_store.cc:269] 0806 20:35:20.992236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.005193 (+ 12957us) service_pool.cc:227] Handling call
0806 20:35:23.358985 (+2353792us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54001081d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:23.359087 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":801195392}
I20260806 20:35:23.372356 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:23.375658 14549 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:23.377256 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:23.178166 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1516) took 2644 ms (client timeout 1878 ms). Trace:
W20260806 20:35:23.389645 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 1.7868e+06us
I20260806 20:35:23.383098 15184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 407) took 1987 ms. Trace:
W20260806 20:35:23.268168 16475 log.cc:924] Time spent T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
W20260806 20:35:23.276093 16469 log.cc:924] Time spent T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
W20260806 20:35:23.393718 14596 connection.cc:232] RPC call timeout handler was delayed by -1.6617414280190133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20260806 20:35:23.395011 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:23.354882 14546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1500) took 2993 ms (client timeout 1565 ms). Trace:
I20260806 20:35:23.399751 14549 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:23.181963 16217 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.393340 15184 rpcz_store.cc:276] 0806 20:35:21.395519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.395724 (+ 205us) service_pool.cc:227] Handling call
0806 20:35:23.380405 (+1984681us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:23.380691 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:23.382666 (+ 1975us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:23.382704 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:23.382762 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:23.382808 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.382882 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:23.383022 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2965760}
I20260806 20:35:23.217206 16397 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:23.386363 16429 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:23.179728 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:23.233446 16151 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:23.381287 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 429011us
W20260806 20:35:23.413719 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 267795us
I20260806 20:35:23.421273 14442 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:23.251232 16477 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.361598 16421 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260806 20:35:23.271059 16483 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:23.424672 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 241605us
I20260806 20:35:23.425544 14442 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:23.163672 16462 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.436805 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:23.442085 14442 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:23.391717 14475 rpcz_store.cc:269] 0806 20:35:20.533694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.537687 (+ 3993us) service_pool.cc:227] Handling call
0806 20:35:23.177928 (+2640241us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540030f054. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:23.178070 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443126016}
I20260806 20:35:23.443885 16423 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.379108 16357 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.444748 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1602) took 1739 ms. Trace:
I20260806 20:35:23.438966 14549 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:23.446661 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:23.445698 14630 rpcz_store.cc:276] 0806 20:35:21.699789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.700105 (+ 316us) service_pool.cc:227] Handling call
0806 20:35:21.738059 (+ 37954us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.738315 (+ 256us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:21.758159 (+ 19844us) log.cc:842] Serialized 14 byte log entry
0806 20:35:23.438928 (+1680769us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:23.438987 (+ 59us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:23.439044 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.439105 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:23.439207 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1562696,"prepare.queue_time_us":23034,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":1569705,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":117721,"threads_started":1,"wal-append.queue_time_us":228934,"wal-append.run_cpu_time_us":1329,"wal-append.run_wall_time_us":1093928}
I20260806 20:35:23.430721 16397 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.444919 16462 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.447736 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:23.451412 14549 raft_consensus.cc:2354] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3.
I20260806 20:35:23.407918 16429 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:23.450836 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1610) took 1710 ms. Trace:
I20260806 20:35:23.483734 14633 rpcz_store.cc:276] 0806 20:35:21.729361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.850952 (+121591us) service_pool.cc:227] Handling call
0806 20:35:21.859781 (+ 8829us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:23.438898 (+1579117us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:23.438930 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:23.438978 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:23.439038 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.439098 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:23.439245 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1569691,"raft.queue_time_us":1585536,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":155}
W20260806 20:35:23.484957 14477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 207) took 1773 ms (client timeout 1590 ms). Trace:
W20260806 20:35:23.485402 14477 rpcz_store.cc:269] 0806 20:35:21.711651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.759932 (+ 48281us) service_pool.cc:227] Handling call
0806 20:35:23.484749 (+1724817us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b5400231ed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:23.484861 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":799387520}
I20260806 20:35:23.427515 16362 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.440373 16440 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.430820 16151 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:23.178249 16478 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.509269 14625 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260806 20:35:23.518119 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 140903us
I20260806 20:35:23.463764 16483 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:23.522957 14595 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:23.523275 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.518903 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:23.523978 14625 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:23.524274 14633 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:35:23.484411 16261 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.525065 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.526582 14596 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:23.528434 14596 leader_election.cc:398] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Vote denied by peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) with higher term. Message: Invalid argument: T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3.
I20260806 20:35:23.528800 14596 leader_election.cc:401] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 20:35:23.502447 16411 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:23.504544 16354 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:23.506076 16344 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.529521 14625 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:23.506181 16295 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:23.502502 16423 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:23.488922 16429 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:23.508545 16296 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:23.515794 16414 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:23.514850 16384 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:23.522768 16483 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:23.514984 16362 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.531523 16261 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.536679 16295 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.528515 16391 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.525166 16440 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.543932 16464 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:23.554486 16442 log.cc:924] Time spent T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 1.394s user 0.006s sys 0.000s
I20260806 20:35:23.542639 16423 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.440990 16477 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.560328 16478 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.561213 16440 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:23.401815 14546 rpcz_store.cc:269] 0806 20:35:20.361338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.382171 (+ 20833us) service_pool.cc:227] Handling call
0806 20:35:23.354567 (+2972396us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54000b2cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:23.354755 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1096600320}
W20260806 20:35:23.599128 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 156866us
W20260806 20:35:23.601089 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 78097.7us
W20260806 20:35:23.506556 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 70986.3us
I20260806 20:35:23.602716 14595 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:23.605629 14595 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:23.605841 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:23.492547 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:23.531847 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:23.613389 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 85495.4us
I20260806 20:35:23.613173 14628 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:23.516369 16049 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260806 20:35:23.635628 16049 consensus_queue.cc:1035] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 20:35:23.641163 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.528716 16151 raft_consensus.cc:686] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:23.643483 16285 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:23.644508 16151 consensus_queue.cc:237] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.645390 16295 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:23.645488 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:23.646469 16295 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:23.647255 16295 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:23.621763 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:23.543835 16296 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.549938 16391 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.537662 16354 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.653126 16296 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:23.654048 16296 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.647928 16295 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:23.654832 16261 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:23.655622 16261 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:23.655395 16295 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.655830 16354 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.658571 16309 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:23.627483 14521 leader_election.cc:334] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.683s (SENT)
I20260806 20:35:23.659479 16309 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.660787 14521 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:23.661228 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1440) took 2646 ms. Trace:
I20260806 20:35:23.661700 14470 rpcz_store.cc:276] 0806 20:35:21.014831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.079497 (+ 64666us) service_pool.cc:227] Handling call
0806 20:35:21.079791 (+ 294us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:21.097603 (+ 17812us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.660467 (+2562864us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b54003f5cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:23.660932 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24826240}
I20260806 20:35:23.662874 16309 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.663695 16329 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:23.532976 16338 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:23.665490 16338 raft_consensus.cc:686] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:23.659415 16362 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.668226 14473 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:35:23.537602 16344 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.546134 16384 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.608443 14546 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:23.608385 14595 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:23.703130 16423 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.704063 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.705060 16384 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.705621 14628 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:23.704793 14372 catalog_manager.cc:5495] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 2 to 3, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:23.706871 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:23.708355 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:23.725131 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 112100us
I20260806 20:35:23.654904 16296 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.667249 16338 consensus_queue.cc:237] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.734128 16295 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:23.737690 14596 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.565s (SENT) [suppressed 9 similar messages]
I20260806 20:35:23.752244 16486 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:23.752805 16399 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:23.753119 16465 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:23.756531 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:23.762094 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:23.762431 16344 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:23.763211 14546 raft_consensus.cc:1261] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:23.535768 16411 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.744326 16404 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.803264 16186 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:23.804800 16404 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.717140 16329 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.805084 14372 catalog_manager.cc:5495] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:23.813047 16477 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.813720 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1336) took 2123 ms. Trace:
I20260806 20:35:23.814370 16423 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.814724 14633 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:23.815377 16423 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.816006 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.816877 14633 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:23.810034 14628 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:23.800740 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1514) took 2594 ms. Trace:
I20260806 20:35:23.829109 14547 rpcz_store.cc:276] 0806 20:35:21.205983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.206185 (+ 202us) service_pool.cc:227] Handling call
0806 20:35:21.206558 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.856329 (+649771us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.800059 (+1943730us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54001fc2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:23.800506 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1324049664}
I20260806 20:35:23.830088 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 213) took 1896 ms. Trace:
I20260806 20:35:23.838771 14372 catalog_manager.cc:5495] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:23.814164 14555 rpcz_store.cc:276] 0806 20:35:21.690678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.690788 (+ 110us) service_pool.cc:227] Handling call
0806 20:35:21.691043 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.865503 (+174460us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.813176 (+1947673us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54003e80d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:23.813542 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":383077248}
I20260806 20:35:23.845155 16285 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:23.856276 16404 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.830793 14632 rpcz_store.cc:276] 0806 20:35:21.933593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.938163 (+ 4570us) service_pool.cc:227] Handling call
0806 20:35:21.938438 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:23.370179 (+1431741us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.829335 (+459156us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002bedd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:23.829776 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2642552576}
I20260806 20:35:23.884904 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.886462 14625 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:23.891465 15150 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:23.814172 16494 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.897933 14633 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:23.901119 16286 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.901618 15181 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 457) took 2112 ms. Trace:
I20260806 20:35:23.901772 16286 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.902052 15181 rpcz_store.cc:276] 0806 20:35:21.788981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.801717 (+ 12736us) service_pool.cc:227] Handling call
0806 20:35:23.838536 (+2036819us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593b7c 00007fb9b059373b 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:23.838889 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:23.892961 (+ 54072us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:23.892991 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:23.894951 (+ 1960us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:23.901110 (+ 6159us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.901191 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:23.901341 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14651136}
I20260806 20:35:23.814364 16492 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.904702 14372 catalog_manager.cc:5495] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 2 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:23.859792 16477 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.890180 16421 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:23.817763 16465 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.913017 16296 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.915249 16397 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.899834 16384 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:23.913794 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.896323 16391 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.910888 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:23.918962 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 122770us
W20260806 20:35:23.915078 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 183784us
I20260806 20:35:23.917701 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:23.921867 14547 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:23.923761 16296 consensus_queue.cc:1035] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:23.924347 16285 consensus_queue.cc:1046] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:23.925467 16285 raft_consensus.cc:3032] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:23.922644 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 212) took 1329 ms. Trace:
I20260806 20:35:23.925815 16285 raft_consensus.cc:729] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:23.927116 16285 consensus_queue.cc:260] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.928951 16285 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:23.907526 16286 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:23.931768 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 323741us
I20260806 20:35:23.934708 14633 raft_consensus.cc:2396] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:23.905328 16492 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.936910 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.926220 14471 rpcz_store.cc:276] 0806 20:35:22.593355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:22.593580 (+ 225us) service_pool.cc:227] Handling call
0806 20:35:22.593849 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:22.594029 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:22.605027 (+ 10998us) log.cc:842] Serialized 14 byte log entry
0806 20:35:23.698659 (+1093632us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b18003585a0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a57b 00007fb9b34fd88a
0806 20:35:23.921296 (+222637us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:23.921511 (+ 215us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:23.921790 (+ 279us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.921935 (+ 145us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:23.922226 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10813,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":2898,"raft.queue_time_us":258,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":232,"spinlock_wait_cycles":4061568,"thread_start_us":211985,"threads_started":1}
I20260806 20:35:23.938953 15946 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:23.940387 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.941126 14633 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:23.942023 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.942989 14633 raft_consensus.cc:2396] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:23.943895 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.944617 16423 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:23.944705 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:23.940567 16425 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:23.946009 16425 raft_consensus.cc:686] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:23.947386 14633 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:23.946242 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:23.939479 16477 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.950057 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.938953 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1603) took 2208 ms. Trace:
I20260806 20:35:23.935259 16286 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:23.972071 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:23.977056 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:23.960022 14372 catalog_manager.cc:5495] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:23.936544 16421 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.984170 16286 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.947176 16425 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:23.998117 16421 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:23.999472 16067 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.002527 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:23.987555 16261 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) with higher term. Message: Invalid argument: T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3.
I20260806 20:35:24.019532 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:23.985908 16397 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:23.959426 16492 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.023295 14546 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:23.913095 16494 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:24.039749 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 91890.3us
I20260806 20:35:24.040853 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:24.002815 16477 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:24.046228 16497 log.cc:924] Time spent T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260806 20:35:24.045997 16397 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:23.979606 14624 rpcz_store.cc:276] 0806 20:35:21.701331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.720393 (+ 19062us) service_pool.cc:227] Handling call
0806 20:35:23.367884 (+1647491us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:23.368232 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:23.368658 (+ 426us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:23.525546 (+156888us) log.cc:842] Serialized 14 byte log entry
0806 20:35:23.787453 (+261907us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:23.800917 (+ 13464us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:23.849576 (+ 48659us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.895627 (+ 46051us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:23.909253 (+ 13626us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":173866,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":2024,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":36040,"spinlock_wait_cycles":11551104,"thread_start_us":336069,"threads_started":2,"wal-append.queue_time_us":365113}
I20260806 20:35:24.090932 14548 raft_consensus.cc:2379] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:24.047437 16329 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.048414 16122 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:24.052199 16364 raft_consensus.cc:3032] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:24.052845 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:24.053395 16483 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.506s user 0.009s sys 0.006s
I20260806 20:35:24.037007 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.054651 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.055426 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.055723 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.043938 16261 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:24.056866 14632 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:24.014098 14372 catalog_manager.cc:5495] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 2 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:24.031466 16465 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.068041 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 215) took 1455 ms. Trace:
I20260806 20:35:24.071151 15946 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:24.071566 14625 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:24.072629 16286 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.074229 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1516) took 2649 ms. Trace:
W20260806 20:35:24.084709 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 154652us
I20260806 20:35:24.091543 14628 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:24.096540 14479 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:24.064841 16399 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:24.099948 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.104166 14476 rpcz_store.cc:276] 0806 20:35:22.612065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:22.612188 (+ 123us) service_pool.cc:227] Handling call
0806 20:35:22.612441 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:22.968578 (+356137us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:22.968609 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:22.968646 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:22.968686 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.067523 (+1098837us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54003c6254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:24.067603 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:24.067861 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":818545920}
I20260806 20:35:24.117654 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.119093 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.115926 16338 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:24.119843 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.120069 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1268) took 2900 ms. Trace:
I20260806 20:35:24.119340 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.113708 14552 rpcz_store.cc:276] 0806 20:35:21.424439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.424630 (+ 191us) service_pool.cc:227] Handling call
0806 20:35:21.424986 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.484482 (+ 59496us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.073658 (+2589176us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b54000866d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:24.074027 (+ 369us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61792640}
I20260806 20:35:24.093412 16364 raft_consensus.cc:729] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:24.091106 16492 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:24.124984 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.129971 14632 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:24.136873 14628 raft_consensus.cc:2379] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 2: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:24.134449 14471 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
W20260806 20:35:24.158424 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 117786us
I20260806 20:35:24.137974 14372 catalog_manager.cc:5495] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 2 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:24.152032 14521 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:24.161485 14476 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:24.139539 16492 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.161597 16408 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:24.142025 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.134521 16465 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:24.166162 16492 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.165899 16465 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.145211 16411 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:24.119719 16483 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.168706 16263 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:24.169520 15184 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:24.162554 16408 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.170543 16483 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.168365 16465 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.172503 16408 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.173398 16397 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.174819 14478 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:24.176994 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.193497 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.120525 14629 rpcz_store.cc:276] 0806 20:35:21.195694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.195863 (+ 169us) service_pool.cc:227] Handling call
0806 20:35:21.196325 (+ 462us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:21.196787 (+ 462us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:21.196829 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:21.197394 (+ 565us) log.cc:842] Serialized 52 byte log entry
0806 20:35:21.457248 (+259854us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:21.471491 (+ 14243us) log.cc:842] Serialized 14 byte log entry
0806 20:35:23.371488 (+1899997us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b1800094a80. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05424ca 00007fb9b9c23f99 00007fb9b0406990 00007fb9b05c8864 00007fb9b05c8155 00007fb9b34eb3d0 00007fb9b34fd576 00007fb9b350e0d7 00007fb9b350e058
0806 20:35:23.857844 (+486356us) log.cc:842] Serialized 14 byte log entry
0806 20:35:24.090340 (+232496us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.090499 (+ 159us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:24.090634 (+ 135us) raft_consensus.cc:1671] finished
0806 20:35:24.090649 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:24.095565 (+ 4916us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":182836,"prepare.queue_time_us":196174,"prepare.run_cpu_time_us":762,"prepare.run_wall_time_us":6150,"raft.queue_time_us":2107218,"raft.run_cpu_time_us":528,"raft.run_wall_time_us":35561,"spinlock_wait_cycles":7601536,"thread_start_us":837275,"threads_started":3,"wal-append.queue_time_us":363309,"wal-append.run_cpu_time_us":11951,"wal-append.run_wall_time_us":1222776}
I20260806 20:35:24.195035 14632 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 1.
I20260806 20:35:24.126156 16122 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:24.203822 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 51944.2us
I20260806 20:35:24.147330 16410 raft_consensus.cc:3032] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:24.206426 14628 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:24.145038 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.158295 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.155354 16424 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.332769 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.349726 14630 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:24.217836 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.292521 14552 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:24.301273 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1337) took 2607 ms. Trace:
I20260806 20:35:24.153504 16357 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:24.309943 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1284) took 2536 ms. Trace:
W20260806 20:35:24.293710 16429 consensus_meta.cc:287] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.745s user 0.013s sys 0.003s
I20260806 20:35:24.271034 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.270955 14595 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:24.134419 14546 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:24.342133 16410 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:24.305961 16397 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:24.345213 16122 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.326520 14521 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:24.376922 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 117438us
W20260806 20:35:24.314698 16186 consensus_meta.cc:287] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.507s user 0.013s sys 0.000s
I20260806 20:35:24.378334 14521 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:24.379179 16492 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:24.379925 16492 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:24.380270 16492 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:24.380679 16492 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.384855 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.385903 14471 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:24.354863 14627 rpcz_store.cc:276] 0806 20:35:21.773736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.876680 (+102944us) service_pool.cc:227] Handling call
0806 20:35:21.877143 (+ 463us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:23.515438 (+1638295us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:23.515479 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:23.521198 (+ 5719us) log.cc:842] Serialized 26 byte log entry
0806 20:35:24.309012 (+787814us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54001507a0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b3483028 00007fb9b34f3a7b 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c
0806 20:35:24.309138 (+ 126us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:24.309210 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.309305 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:24.309384 (+ 79us) raft_consensus.cc:1671] finished
0806 20:35:24.309404 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:24.309719 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1642608,"raft.queue_time_us":1638432,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":5477,"spinlock_wait_cycles":573630336,"thread_start_us":73875,"threads_started":1,"wal-append.queue_time_us":36071}
I20260806 20:35:24.388931 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.389678 14627 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:24.390609 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.338910 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.405622 16429 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:24.319005 16501 log.cc:924] Time spent T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.440s user 0.000s sys 0.002s
I20260806 20:35:24.415932 16384 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:24.400292 14633 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:24.392788 14624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
mode: GRACEFUL
new_leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
from {username='slave'} at 127.0.0.1:47804
I20260806 20:35:24.350881 14554 rpcz_store.cc:276] 0806 20:35:21.693282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.693399 (+ 117us) service_pool.cc:227] Handling call
0806 20:35:21.693715 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.858910 (+165195us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.270539 (+2411629us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54000a8a54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:24.301015 (+ 30476us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378632576}
I20260806 20:35:24.300571 16287 raft_consensus.cc:3032] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:24.248754 16494 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.557641 14627 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:24.558261 14629 raft_consensus.cc:2396] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:24.218624 14372 catalog_manager.cc:5495] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 2 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:24.306861 15180 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:24.151367 16415 consensus_queue.cc:1046] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.203s
I20260806 20:35:24.359426 14632 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:24.218799 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.373317 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.374740 16410 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.362744 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.376844 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.387916 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:24.467911 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 89647.4us
W20260806 20:35:24.301225 16480 log.cc:924] Time spent T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 1.224s user 0.000s sys 0.008s
I20260806 20:35:24.217916 16408 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:24.554442 16429 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.560418 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.570492 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.561761 16067 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:24.557150 16287 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:24.573376 15181 raft_consensus.cc:2396] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:35:24.573163 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:24.574190 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 450) took 3659 ms. Trace:
I20260806 20:35:24.575364 16206 raft_consensus.cc:3032] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
W20260806 20:35:24.576853 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 383858us
I20260806 20:35:24.357043 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.572466 16415 consensus_queue.cc:1225] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer df1245ecd2034e11ae6cf688c1c10a4d log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:35:24.577491 14372 catalog_manager.cc:5495] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 2 to 3, leader changed from 148bf2a11dba423ab364371f0458e871 (127.12.248.132) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:24.567307 16285 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:24.586118 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.602453 16287 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.605626 16273 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:24.605911 14546 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:24.580796 16408 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:24.625191 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.626253 14629 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:24.629194 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1624) took 1002 ms. Trace:
I20260806 20:35:24.629719 14478 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:24.625820 14632 raft_consensus.cc:1226] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:24.629624 14629 rpcz_store.cc:276] 0806 20:35:23.624472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.571392 (+946920us) service_pool.cc:227] Handling call
0806 20:35:24.584487 (+ 13095us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:24.627090 (+ 42603us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.627254 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:24.627458 14628 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:24.630743 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1623) took 1006 ms. Trace:
I20260806 20:35:24.631116 14632 rpcz_store.cc:276] 0806 20:35:23.624017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.571455 (+947438us) service_pool.cc:227] Handling call
0806 20:35:24.607228 (+ 35773us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:24.630492 (+ 23264us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.630690 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:24.632668 16186 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:24.606441 16429 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:24.625303 16410 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.623857 16397 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.640504 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.627938 16408 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.641184 16031 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:24.637667 14552 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:24.649770 16186 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.650599 14474 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:24.651696 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.652105 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.652491 14476 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:24.652802 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:24.603715 15180 rpcz_store.cc:276] 0806 20:35:20.914541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.915404 (+ 863us) service_pool.cc:227] Handling call
0806 20:35:21.852865 (+937461us) spinlock_profiling.cc:243] Waited 4.22 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:24.306194 (+2453329us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593b7c 00007fb9b059373b 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:24.306518 (+ 324us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:24.573886 (+267368us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.574120 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38450688}
I20260806 20:35:24.632923 16067 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.654585 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.655200 14476 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:24.655561 14546 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:24.655690 16067 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.652947 14479 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:24.653590 14477 raft_consensus.cc:2362] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4. Re-sending same reply.
I20260806 20:35:24.656603 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.658185 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.658510 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.610378 16206 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:24.660981 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:24.625011 16399 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:24.628070 16273 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:24.666638 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 389259us
I20260806 20:35:24.667228 16399 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.666253 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.669509 14552 raft_consensus.cc:2396] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:24.643317 16287 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:24.622925 16285 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.672475 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.675109 14521 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:24.680934 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.652351 16186 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.671823 14372 catalog_manager.cc:5495] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 2 to 3, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:24.698117 16280 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:24.698906 16280 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.700851 16067 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
W20260806 20:35:24.701033 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 63133.7us
I20260806 20:35:24.701458 16067 raft_consensus.cc:686] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:24.702714 16494 consensus_queue.cc:1046] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:35:24.702693 16067 consensus_queue.cc:237] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.704406 16494 consensus_queue.cc:1046] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:24.704744 16423 consensus_queue.cc:1046] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 20:35:24.705091 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 76754us
I20260806 20:35:24.706038 14595 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:24.705794 16494 raft_consensus.cc:3032] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:24.707044 14518 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:24.707218 16494 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:24.710332 16468 log.cc:924] Time spent T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 1.977s user 0.007s sys 0.013s
I20260806 20:35:24.709729 16206 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.710168 16494 consensus_queue.cc:260] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.711472 16206 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.711627 16494 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:24.711943 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.713284 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1354) took 2095 ms. Trace:
I20260806 20:35:24.689692 16411 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:24.713771 14551 rpcz_store.cc:276] 0806 20:35:22.617396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:22.728521 (+111125us) service_pool.cc:227] Handling call
0806 20:35:22.728891 (+ 370us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:22.730634 (+ 1743us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:22.730671 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:22.780178 (+ 49507us) log.cc:842] Serialized 26 byte log entry
0806 20:35:23.018750 (+238572us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:23.018807 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:23.018891 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:24.197797 (+1178906us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:24.712822 (+515025us) raft_consensus.cc:1671] finished
0806 20:35:24.712845 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:24.713100 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1302,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":197,"spinlock_wait_cycles":5032320,"thread_start_us":63337,"threads_started":1,"wal-append.queue_time_us":262417}
I20260806 20:35:24.713243 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.715636 14470 raft_consensus.cc:1261] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:24.710621 16384 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:24.716181 14547 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:24.668315 16273 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.730948 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:24.673746 14475 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:24.741043 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.659564 16031 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.754165 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.690598 16429 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.748204 16384 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.760924 14471 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:24.762004 16031 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.762348 16285 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.764175 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.778565 14547 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:24.766521 16280 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:24.767387 16472 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:35:24.770948 16461 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:24.782269 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.783004 14629 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:24.783877 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:24.784634 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 104227us
I20260806 20:35:24.771961 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 217) took 1320 ms. Trace:
I20260806 20:35:24.787858 14631 rpcz_store.cc:276] 0806 20:35:23.451040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.505905 (+ 54865us) service_pool.cc:227] Handling call
0806 20:35:23.523275 (+ 17370us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:23.670967 (+147692us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:23.671723 (+ 756us) log.cc:842] Serialized 14 byte log entry
0806 20:35:23.745630 (+ 73907us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:23.754344 (+ 8714us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:23.757046 (+ 2702us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.771552 (+1014506us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400161554. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:24.771597 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:24.771785 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25434,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":131,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":7791,"spinlock_wait_cycles":338243968,"thread_start_us":70840,"threads_started":1,"wal-append.queue_time_us":366458}
I20260806 20:35:24.791637 14546 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:24.792532 14442 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:24.793283 16408 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.793592 16031 consensus_queue.cc:1046] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:35:24.791102 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.794562 16031 raft_consensus.cc:3032] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:24.794955 16031 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:24.796066 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.793620 16280 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:24.798100 16280 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:24.798590 16280 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:24.798835 14595 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:24.800431 16419 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:24.800891 14631 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:24.797201 16471 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:24.801041 16419 raft_consensus.cc:686] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:24.801705 16471 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.802277 16419 consensus_queue.cc:237] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.797317 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:24.801004 16399 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:24.774492 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.807188 14632 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:24.795920 16031 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.754700 14372 catalog_manager.cc:5495] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 2 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:24.805014 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.805677 14629 raft_consensus.cc:2379] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:24.804316 16471 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:24.790130 16492 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.815001 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.818322 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.820017 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.821535 14372 catalog_manager.cc:5495] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 3, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:24.861153 14632 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:24.862854 14554 raft_consensus.cc:2379] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:24.869493 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.872444 14479 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:35:24.879513 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:24.882963 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 133939us
I20260806 20:35:24.883035 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.881330 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.871066 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.892995 14632 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:24.892108 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 136468us
I20260806 20:35:24.882061 16031 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.891595 14596 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:24.896885 14547 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:35:24.913081 14442 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:24.916567 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:24.917200 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 293894us
W20260806 20:35:24.918022 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 111014us
I20260806 20:35:24.918171 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 205) took 3360 ms. Trace:
I20260806 20:35:24.918610 14626 rpcz_store.cc:276] 0806 20:35:21.558009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:21.558153 (+ 144us) service_pool.cc:227] Handling call
0806 20:35:21.603620 (+ 45467us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:21.789928 (+186308us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.917768 (+3127840us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54003bef54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:24.917981 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":298641792}
I20260806 20:35:24.920015 14626 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:24.920397 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1291) took 1156 ms. Trace:
W20260806 20:35:24.920527 14469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 4.037s user 0.015s sys 0.065s
W20260806 20:35:24.921000 14469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.137s user 0.015s sys 0.065s
I20260806 20:35:24.921482 15150 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:24.921939 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 211) took 2353 ms. Trace:
I20260806 20:35:24.922348 14473 rpcz_store.cc:276] 0806 20:35:22.568534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:22.568670 (+ 136us) service_pool.cc:227] Handling call
0806 20:35:23.667420 (+1098750us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:23.668713 (+ 1293us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.921730 (+1253017us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003f5cd0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:24.921886 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2462485888}
I20260806 20:35:24.925673 16122 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:24.926111 16122 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:24.926489 16122 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:24.926892 14596 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:24.927037 14624 raft_consensus.cc:595] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Received request to transfer leadership to TS 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:24.927559 14624 raft_consensus.cc:603] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:24.927838 16411 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:24.928078 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47804 (request call id 8) took 1244 ms. Trace:
I20260806 20:35:24.928267 16411 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:24.916741 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:24.929441 16514 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:24.920604 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1295) took 1072 ms. Trace:
I20260806 20:35:24.933558 16434 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:24.935880 16488 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:35:24.935997 14372 catalog_manager.cc:5495] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 2, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 2 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:24.936740 16122 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:24.938848 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1631) took 1240 ms. Trace:
I20260806 20:35:24.887614 16397 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:24.894719 16357 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.527s user 0.007s sys 0.004s
I20260806 20:35:24.925712 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:24.920816 14629 rpcz_store.cc:276] 0806 20:35:23.763994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.908125 (+1144131us) service_pool.cc:227] Handling call
0806 20:35:24.917380 (+ 9255us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:24.920034 (+ 2654us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:24.920069 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:24.920131 (+ 62us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:24.920164 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.920223 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:24.920324 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:24.935927 16516 log.cc:924] Time spent T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.271s user 0.000s sys 0.002s
I20260806 20:35:24.947707 14626 rpcz_store.cc:276] 0806 20:35:23.858728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.919513 (+1060785us) service_pool.cc:227] Handling call
0806 20:35:24.919843 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:24.920369 (+ 526us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.920540 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:24.950469 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:24.984264 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 82522.5us
I20260806 20:35:24.948413 16399 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:24.951942 16434 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.957005 14627 rpcz_store.cc:276] 0806 20:35:23.698043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.588861 (+890818us) service_pool.cc:227] Handling call
0806 20:35:24.620177 (+ 31316us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:24.638532 (+ 18355us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:24.638569 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:24.639043 (+ 474us) log.cc:842] Serialized 26 byte log entry
0806 20:35:24.727990 (+ 88947us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:24.879204 (+151214us) log.cc:842] Serialized 14 byte log entry
0806 20:35:24.881472 (+ 2268us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.881566 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:24.938616 (+ 57050us) raft_consensus.cc:1671] finished
0806 20:35:24.938638 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:24.938771 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1872,"raft.queue_time_us":7317,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":3229,"thread_start_us":220774,"threads_started":2,"wal-append.queue_time_us":233063}
W20260806 20:35:24.951977 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 500956bd1a3846a3a7b16c51ddecbc44 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:24.946897 16411 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:25.000625 14356 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48911.3us
I20260806 20:35:24.947573 16514 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:25.003439 14596 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:24.953765 16408 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.022488 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:25.025802 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 106406us
I20260806 20:35:25.025669 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.022745 14629 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:25.030077 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1301) took 1069 ms. Trace:
I20260806 20:35:25.030383 14629 rpcz_store.cc:276] 0806 20:35:23.965567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.999992 (+1034425us) service_pool.cc:227] Handling call
0806 20:35:25.022558 (+ 22566us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.029800 (+ 7242us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.030013 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:24.914768 16285 consensus_queue.cc:1046] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260806 20:35:25.022253 14372 catalog_manager.cc:5495] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff reported cstate change: term changed from 1 to 3, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:25.037988 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:25.038462 16533 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:25.038800 16122 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:25.039623 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1535) took 1112 ms. Trace:
I20260806 20:35:25.039737 14629 raft_consensus.cc:1261] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:25.040004 14554 rpcz_store.cc:276] 0806 20:35:23.927322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.643146 (+715824us) service_pool.cc:227] Handling call
0806 20:35:25.039431 (+396285us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b540018e7d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:25.039572 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":413023104}
I20260806 20:35:25.041246 16486 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:25.041944 16486 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.042943 14518 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:25.042831 16280 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.044294 14518 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:25.044580 14477 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:25.045571 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:25.041301 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.046164 16474 raft_consensus.cc:2741] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:25.046986 14554 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:25.047053 16474 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:25.047942 16474 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:25.047994 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.038841 14549 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:25.039144 16533 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:24.961889 14631 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:25.048192 14518 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:24.999806 16399 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:25.172114 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54486 (request call id 1220) took 1487 ms. Trace:
W20260806 20:35:25.172370 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 124216us
I20260806 20:35:25.092015 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:24.943733 14624 rpcz_store.cc:276] 0806 20:35:23.683620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.392614 (+708994us) service_pool.cc:227] Handling call
0806 20:35:24.926812 (+534198us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b844c330 00007fb9b8437dc9 00007fb9b843768c 00007fb9b1b2ad04
0806 20:35:24.927947 (+ 1135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6996352}
I20260806 20:35:25.107317 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1292) took 1343 ms. Trace:
I20260806 20:35:25.184849 14551 raft_consensus.cc:1226] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:25.157792 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 250) took 1722 ms. Trace:
I20260806 20:35:25.093214 14477 raft_consensus.cc:1226] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:25.075008 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:25.044402 14470 raft_consensus.cc:2379] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:25.073141 14626 raft_consensus.cc:1226] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:25.028720 16514 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:25.068780 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:25.223347 14630 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:25.071341 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:25.061043 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 57933.6us
I20260806 20:35:25.103448 15153 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:25.110450 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 83472.8us
I20260806 20:35:25.091841 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:25.085500 14547 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:25.172648 14555 rpcz_store.cc:276] 0806 20:35:23.670118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.842036 (+171918us) service_pool.cc:227] Handling call
0806 20:35:25.107601 (+1265565us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54000870d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b8438c6e 00007fb9b1b2f807 00007fb9b1b2f792 00007fb9b1b2f6c1 00007fb9b1b2f63c 00007fb9b1b2e8f2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:25.156834 (+ 49233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":266692352}
I20260806 20:35:25.053654 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:25.184191 14633 rpcz_store.cc:276] 0806 20:35:23.764288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.951795 (+187507us) service_pool.cc:227] Handling call
0806 20:35:25.107044 (+1155249us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b54002c2254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:25.107182 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88355200}
I20260806 20:35:25.216779 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1529) took 1259 ms. Trace:
I20260806 20:35:25.183471 14554 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:25.217528 14549 rpcz_store.cc:276] 0806 20:35:23.435218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.715085 (+279867us) service_pool.cc:227] Handling call
0806 20:35:25.038537 (+1323452us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593c6c 00007fb9b0593bd6 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:25.038663 (+ 126us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.157210 (+118547us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.157670 (+ 460us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4776064}
I20260806 20:35:25.086833 16434 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.035116 16492 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:25.053587 16179 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:25.062467 16280 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.030588 16525 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:25.247591 14476 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:25.247766 14633 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:25.225600 14548 rpcz_store.cc:276] 0806 20:35:23.924921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.945954 (+ 21033us) service_pool.cc:227] Handling call
0806 20:35:25.184265 (+1238311us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400086bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:25.184486 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":287489280}
W20260806 20:35:25.227780 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 303356us
I20260806 20:35:25.249953 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:25.107981 16122 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:25.103631 16429 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:25.054896 16447 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:25.165872 16338 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:25.174340 16399 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:25.233480 14518 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:25.236675 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1528) took 1605 ms. Trace:
I20260806 20:35:25.264549 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:35:25.264669 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:25.268744 16455 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:25.269944 16338 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:25.269706 16455 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.270479 16361 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:25.270915 16338 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.269306 16122 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.273257 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.273645 16361 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.274580 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.275163 14478 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:25.275497 16525 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.276147 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:25.273175 16122 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.276837 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:25.276988 16122 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:25.277735 15179 raft_consensus.cc:2354] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:25.277614 16122 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.279850 16122 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:25.044353 16486 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:25.281221 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:25.246949 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 243599us
W20260806 20:35:25.249321 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 145976us
I20260806 20:35:25.189733 16408 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:25.105552 16483 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.125s
I20260806 20:35:25.255800 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.253801 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:25.258082 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:25.287081 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 213852us
I20260806 20:35:25.254000 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:25.165217 16364 consensus_queue.cc:260] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.097311 16456 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.136s
I20260806 20:35:25.288367 16408 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.301152 16408 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:25.301263 16525 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:25.302673 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.303876 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.304431 14473 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:25.308424 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.309089 14473 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:25.309521 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.310655 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.311188 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.256815 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.304838 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.311648 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.260290 14442 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:25.268260 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:35:25.286648 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.329311 14548 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:25.264256 14553 rpcz_store.cc:276] 0806 20:35:23.631214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.763259 (+132045us) service_pool.cc:227] Handling call
0806 20:35:25.223483 (+1460224us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:25.223724 (+ 241us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.236294 (+ 12570us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:25.236328 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:25.236382 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:25.236416 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.236500 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:25.236616 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5290752}
I20260806 20:35:25.255445 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1363) took 1219 ms. Trace:
I20260806 20:35:25.298619 16361 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:25.311435 15187 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:25.312053 15182 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:25.312686 14479 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:25.262094 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:25.306082 14629 raft_consensus.cc:1261] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:25.314144 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.306836 15153 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:25.260679 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:25.071612 16397 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.162s
I20260806 20:35:25.307296 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.278837 16280 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:25.256901 16488 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:25.324273 14476 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:25.339699 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.289907 14521 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:25.332705 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 108542us
I20260806 20:35:25.269284 16524 raft_consensus.cc:2741] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 20:35:25.339282 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.341436 15184 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:25.352447 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.379114 14479 raft_consensus.cc:2379] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:25.353813 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.377801 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.377218 14476 raft_consensus.cc:1261] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:25.349910 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:25.376149 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.288841 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:25.306453 16364 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:25.289825 16122 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:25.305142 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.380178 14444 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:25.396775 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 173216us
W20260806 20:35:25.400184 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 110345us
I20260806 20:35:25.404641 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:25.406136 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:25.419137 15187 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:25.306515 16533 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.404623 16166 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:25.411173 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.353200 16488 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.411247 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.427953 14477 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:25.332119 16483 consensus_queue.cc:1046] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260806 20:35:25.379783 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1522) took 4799 ms. Trace:
I20260806 20:35:25.426033 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:25.433314 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.433717 14476 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:25.434055 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:25.434792 14548 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:25.399835 16486 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:25.377990 16361 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:25.419227 14521 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:25.426812 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:25.423352 14595 connection.cc:232] RPC call timeout handler was delayed by -1.2564508550276514s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31220000000000003s for a response
W20260806 20:35:25.436102 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 182059us
I20260806 20:35:25.429966 16488 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:25.439561 14479 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:25.440160 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.442684 16303 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:25.445765 16486 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.427466 16166 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.448769 16166 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:25.424616 16492 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:25.332791 14546 rpcz_store.cc:276] 0806 20:35:24.028761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.656258 (+627497us) service_pool.cc:227] Handling call
0806 20:35:25.230319 (+574061us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54002da5d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:25.241238 (+ 10919us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":308529920}
I20260806 20:35:25.456045 14554 raft_consensus.cc:2379] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 2: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:25.420291 14631 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:25.458920 14521 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:25.460589 14549 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:25.461978 16539 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:25.462721 14518 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:25.463449 16434 consensus_queue.cc:1035] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:25.435829 16122 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.465744 16486 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.466552 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:25.468837 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1290) took 1705 ms. Trace:
I20260806 20:35:25.454420 16338 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.424800 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.434152 16429 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:25.471661 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 121739us
W20260806 20:35:25.443573 16384 consensus_meta.cc:287] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.661s user 0.001s sys 0.018s
I20260806 20:35:25.514102 16384 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:25.476410 14518 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:25.515775 16292 raft_consensus.cc:3032] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:25.516261 16292 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:25.516938 14627 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:25.517303 16486 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:25.517606 16292 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.443320 16525 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:25.439510 16361 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.442713 16402 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:25.578897 16525 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.579828 16525 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:25.435164 14469 rpcz_store.cc:276] 0806 20:35:20.549570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:20.610060 (+ 60490us) service_pool.cc:227] Handling call
0806 20:35:20.728998 (+118938us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:21.527270 (+798272us) spinlock_profiling.cc:243] Waited 5.73 ms on lock 0x7b18002e8400. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b0598f96 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511
0806 20:35:24.574601 (+3047331us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b05ad5fe 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05c8155 00007fb9b34eb3d0
0806 20:35:24.921488 (+346887us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:24.921522 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:24.922132 (+ 610us) log.cc:842] Serialized 26 byte log entry
0806 20:35:25.028642 (+106510us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:25.036202 (+ 7560us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.176229 (+140027us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:25.304304 (+128075us) raft_consensus.cc:1671] finished
0806 20:35:25.304360 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:25.340793 (+ 36433us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":159,"raft.queue_time_us":20801,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":23249152,"thread_start_us":4229833,"threads_started":2,"wal-append.queue_time_us":333733}
I20260806 20:35:25.577706 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1287) took 1852 ms. Trace:
I20260806 20:35:25.495260 14442 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:25.498699 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 44336.4us
I20260806 20:35:25.502808 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:25.499971 14632 rpcz_store.cc:276] 0806 20:35:23.763666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.897143 (+1133477us) service_pool.cc:227] Handling call
0806 20:35:25.160916 (+263773us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.172940 (+ 12024us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:25.172998 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:25.173055 (+ 57us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:25.173105 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.173186 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:25.468551 (+295365us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400051750. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:25.468705 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":489559808}
I20260806 20:35:25.508559 14554 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:25.480495 16474 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:25.519392 14518 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:25.587761 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 68445.3us
I20260806 20:35:25.588496 14518 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:25.518507 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.569662 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1365) took 1336 ms. Trace:
I20260806 20:35:25.501519 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.560161 15187 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:25.607223 14630 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:25.483248 16492 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:25.583532 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1367) took 1017 ms. Trace:
I20260806 20:35:25.584492 15184 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:25.582710 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1537) took 1654 ms. Trace:
W20260806 20:35:25.562077 16494 consensus_meta.cc:287] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.850s user 0.006s sys 0.010s
I20260806 20:35:25.585678 14548 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:25.607318 14553 rpcz_store.cc:276] 0806 20:35:24.183948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.482209 (+1298261us) service_pool.cc:227] Handling call
0806 20:35:25.508233 (+ 26024us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.520042 (+ 11809us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:25.520068 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:25.520109 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:25.520134 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.520189 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:25.520310 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:25.582178 14628 rpcz_store.cc:276] 0806 20:35:23.762454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.631203 (+868749us) service_pool.cc:227] Handling call
0806 20:35:24.648417 (+ 17214us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:24.695981 (+ 47564us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:24.696013 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:24.751103 (+ 55090us) log.cc:842] Serialized 26 byte log entry
0806 20:35:24.960893 (+209790us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:24.960930 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:24.961011 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:25.577461 (+616450us) raft_consensus.cc:1671] finished
0806 20:35:25.577485 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:25.577635 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7078,"raft.queue_time_us":12340,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":8176,"thread_start_us":104174,"threads_started":1,"wal-append.queue_time_us":656797}
I20260806 20:35:25.611724 14444 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:25.622167 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.623024 14548 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
W20260806 20:35:25.519855 16461 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.737s user 0.010s sys 0.003s
I20260806 20:35:25.624183 14596 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:25.614737 14595 leader_election.cc:398] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:25.624840 14595 leader_election.cc:401] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:25.618943 14554 rpcz_store.cc:276] 0806 20:35:24.566308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.491446 (+925138us) service_pool.cc:227] Handling call
0806 20:35:25.494370 (+ 2924us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.583285 (+ 88915us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.583482 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:25.606433 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 124070us
I20260806 20:35:25.578408 15151 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:25.626618 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48297.2us
I20260806 20:35:25.523782 16429 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.633848 14518 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:25.614118 14552 rpcz_store.cc:276] 0806 20:35:23.928571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.713699 (+785128us) service_pool.cc:227] Handling call
0806 20:35:24.714033 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:24.906311 (+192278us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:24.906356 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:24.948469 (+ 42113us) log.cc:842] Serialized 26 byte log entry
0806 20:35:25.139401 (+190932us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:25.139453 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.139571 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:25.582492 (+442921us) raft_consensus.cc:1671] finished
0806 20:35:25.582514 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:25.582651 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":77261,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":379077,"threads_started":2,"wal-append.queue_time_us":623492}
I20260806 20:35:25.560699 16263 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:25.512465 16524 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:25.521273 16338 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:25.587937 16361 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:25.587266 16474 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.560647 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.606036 16436 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:25.615770 15946 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:25.512666 14624 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:25.648613 14553 raft_consensus.cc:1261] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:25.651788 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 67882.2us
I20260806 20:35:25.611068 16402 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.653406 16122 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:25.656024 14444 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:25.656720 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:25.648995 16494 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:25.640908 16429 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:25.497619 16539 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.661437 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:25.666388 16533 consensus_queue.cc:1046] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:25.668378 16338 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.672308 14547 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:25.673734 16361 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.677399 15184 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
W20260806 20:35:25.632352 16411 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.552s user 0.013s sys 0.000s
I20260806 20:35:25.615182 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:25.763350 15821 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260806 20:35:25.683736 14548 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:25.685036 15151 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:25.634495 16461 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:25.639990 16456 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:25.646211 16285 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:25.649118 16474 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:25.611347 16473 consensus_queue.cc:1035] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.169s
I20260806 20:35:25.698292 16514 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.642095 16397 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:35:25.710278 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 54385.2us
I20260806 20:35:25.647919 14546 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:25.569167 16166 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.742448 16402 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:25.746841 16488 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.751971 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1635) took 1010 ms. Trace:
I20260806 20:35:25.698439 15946 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:25.647233 16532 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:25.755079 14554 raft_consensus.cc:1226] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:25.734263 14595 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:25.762910 16411 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.645944 16263 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:25.647432 16524 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:25.764298 16361 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:26.025512 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 357996us
W20260806 20:35:26.000406 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1615) took 2373 ms (client timeout 2130 ms). Trace:
I20260806 20:35:26.478741 16524 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:26.077138 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 426934us
W20260806 20:35:26.139110 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 498626us
I20260806 20:35:25.772821 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:26.483161 14628 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:26.482232 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 797289us
I20260806 20:35:26.481810 14627 rpcz_store.cc:276] 0806 20:35:24.741806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.009311 (+267505us) service_pool.cc:227] Handling call
0806 20:35:25.030340 (+ 21029us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.517672 (+487332us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:25.751678 (+234006us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54002bded4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:25.751898 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1118609152}
I20260806 20:35:26.485471 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 259) took 1434 ms. Trace:
I20260806 20:35:26.486183 15151 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.486088 14554 rpcz_store.cc:276] 0806 20:35:25.050774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.735697 (+684923us) service_pool.cc:227] Handling call
0806 20:35:25.736068 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.484255 (+748187us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.484445 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:25.736289 16424 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.176s user 0.014s sys 0.003s
I20260806 20:35:26.487972 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.809988 16545 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:26.488902 16545 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.169395 16338 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.490104 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.764875 16542 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:26.491099 14554 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:26.369174 16465 raft_consensus.cc:3032] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:26.156757 16539 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.491884 16545 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:26.492127 16465 raft_consensus.cc:729] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:25.945617 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 310054us
I20260806 20:35:26.493757 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:25.764355 16546 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:26.493803 14442 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.495298 14554 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:26.494753 16465 consensus_queue.cc:260] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:26.220023 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 229641us
I20260806 20:35:26.499265 14442 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:26.500324 16566 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:26.500901 16566 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:26.501489 16566 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:26.501024 16567 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:26.503031 16567 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.503579 14442 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:26.478780 14625 rpcz_store.cc:269] 0806 20:35:23.610715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.621049 (+ 10334us) service_pool.cc:227] Handling call
0806 20:35:25.983127 (+2362078us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540039fdd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:25.983345 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":473231744}
I20260806 20:35:26.480993 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1540) took 1590 ms. Trace:
I20260806 20:35:26.506120 14548 rpcz_store.cc:276] 0806 20:35:24.890594 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.633083 (+742489us) service_pool.cc:227] Handling call
0806 20:35:25.650322 (+ 17239us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.480653 (+830331us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.480923 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:25.915330 16455 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.508643 14441 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:26.247704 14629 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.013s sys 0.010s
W20260806 20:35:26.509357 14629 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s user 0.018s sys 0.017s
I20260806 20:35:26.510037 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:25.709352 16494 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:26.513597 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:26.086551 16447 consensus_meta.cc:287] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.823s user 0.009s sys 0.003s
I20260806 20:35:26.517359 14596 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:25.939831 16408 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.638s user 0.007s sys 0.006s
W20260806 20:35:26.063386 16031 consensus_meta.cc:287] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.156s user 0.012s sys 0.001s
I20260806 20:35:26.516942 14625 raft_consensus.cc:2362] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4. Re-sending same reply.
W20260806 20:35:26.509701 16399 consensus_meta.cc:287] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.221s user 0.010s sys 0.024s
I20260806 20:35:26.523094 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:26.512856 16361 raft_consensus.cc:2741] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:35:26.529711 14627 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:25.863447 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:25.765844 15186 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:26.519093 16494 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:26.040411 16157 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:26.506568 15946 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:26.527365 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:26.543646 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:26.520442 16179 consensus_meta.cc:287] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.264s user 0.014s sys 0.000s
I20260806 20:35:26.534336 14521 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:26.545974 14596 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:26.535003 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 255) took 1506 ms. Trace:
I20260806 20:35:26.515021 16424 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:26.522631 16539 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:26.484221 16456 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:26.519323 16542 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.520879 16338 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.531060 16455 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:26.530019 16533 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.554847 14521 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:26.559942 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 50008.6us
I20260806 20:35:26.532411 16415 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:26.536042 16361 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:26.547863 16333 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.563159 14550 raft_consensus.cc:2396] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:35:26.563040 16456 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:26.554821 16402 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:26.546391 15185 raft_consensus.cc:2379] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:26.535553 16157 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:26.567538 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:26.533445 16399 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.586467 16333 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.593454 16399 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.593628 14474 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:26.569164 16539 consensus_queue.cc:1035] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 20:35:26.597061 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:26.597470 14444 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.598584 16263 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:26.598862 14624 raft_consensus.cc:1261] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260806 20:35:26.599123 16525 consensus_meta.cc:287] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.011s user 0.012s sys 0.000s
I20260806 20:35:26.599563 16525 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.600652 14553 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:26.590476 16157 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:26.581493 16524 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.601742 16524 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.603405 16525 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:26.603960 16280 consensus_meta.cc:287] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.159s user 0.013s sys 0.001s
I20260806 20:35:26.606053 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:26.542647 16555 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:26.529050 16408 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.547988 16309 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.549981 16447 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:26.576750 14627 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:26.542493 16031 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:26.580859 16410 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:26.559170 16424 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.569401 16545 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:26.544641 16285 raft_consensus.cc:686] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:26.553966 16567 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.586853 14628 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:26.569504 16455 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:26.573093 16436 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.628108 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.570806 16533 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.570015 16338 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.616192 16399 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:26.680205 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1646) took 1030 ms. Trace:
I20260806 20:35:26.721194 14628 rpcz_store.cc:276] 0806 20:35:25.649654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.735574 (+ 85920us) service_pool.cc:227] Handling call
0806 20:35:26.680135 (+944561us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:26.587728 16415 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.614187 16555 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:26.727911 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:26.728654 14628 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:26.702979 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:26.703152 16455 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:26.619549 16447 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:26.620153 16303 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:26.621763 16410 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:26.678977 16429 consensus_meta.cc:287] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.910s user 0.002s sys 0.010s
I20260806 20:35:26.741413 16429 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.742300 16558 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:26.742970 16558 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:26.711717 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 91133.4us
I20260806 20:35:26.745904 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:26.747874 14444 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.748526 16410 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:26.749718 16338 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:26.750969 16558 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.618327 16309 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.752751 16303 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:26.753314 16303 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:26.753731 16303 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:26.754668 14471 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:26.755599 14555 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:26.619063 14624 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:26.756925 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 238) took 1109 ms. Trace:
I20260806 20:35:26.723963 14521 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:26.757542 15946 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.679543 16031 raft_consensus.cc:686] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:26.759159 16031 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.759775 15946 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.760941 16285 consensus_queue.cc:237] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.704351 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 239) took 1043 ms. Trace:
I20260806 20:35:26.762566 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1575) took 2008 ms. Trace:
I20260806 20:35:26.763020 14470 rpcz_store.cc:276] 0806 20:35:24.754838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.778328 (+ 23490us) service_pool.cc:227] Handling call
0806 20:35:26.762325 (+1983997us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54001a59d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:26.762474 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":540071296}
I20260806 20:35:26.640084 16525 consensus_queue.cc:1035] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.092s
I20260806 20:35:26.764565 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 484) took 1283 ms. Trace:
I20260806 20:35:26.724956 16399 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:26.705084 16411 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:26.710306 16488 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.716733 16166 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.730746 16555 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:26.737998 16447 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:26.738973 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 1.00477e+06us
I20260806 20:35:26.740855 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1647) took 1077 ms. Trace:
I20260806 20:35:26.741892 16474 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.678220 16545 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:26.757443 14624 rpcz_store.cc:276] 0806 20:35:25.647684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.599828 (+952144us) service_pool.cc:227] Handling call
0806 20:35:26.600166 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.756621 (+156455us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.756857 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:26.762771 14625 rpcz_store.cc:276] 0806 20:35:25.661152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.638078 (+976926us) service_pool.cc:227] Handling call
0806 20:35:26.703837 (+ 65759us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.704041 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:26.704066 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:26.704107 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:26.704137 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.704193 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:26.704295 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:26.712193 16436 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.728145 16567 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:26.765020 15185 rpcz_store.cc:276] 0806 20:35:25.481452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.483575 (+ 2123us) service_pool.cc:227] Handling call
0806 20:35:26.739209 (+1255634us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b540037f0d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:26.764443 (+ 25234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1744217344}
I20260806 20:35:26.769155 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 235) took 1342 ms. Trace:
I20260806 20:35:26.797646 14629 rpcz_store.cc:276] 0806 20:35:25.426300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.507708 (+ 81408us) service_pool.cc:227] Handling call
0806 20:35:25.587409 (+ 79701us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:25.623521 (+ 36112us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:25.642009 (+ 18488us) log.cc:842] Serialized 14 byte log entry
0806 20:35:26.543044 (+901035us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:26.550470 (+ 7426us) log.cc:842] Serialized 26 byte log entry
0806 20:35:26.550649 (+ 179us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:26.550687 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.550753 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:26.768602 (+217849us) raft_consensus.cc:1671] finished
0806 20:35:26.768650 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:26.768997 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23682,"prepare.queue_time_us":19190,"raft.queue_time_us":36798,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":38032,"spinlock_wait_cycles":43785088,"thread_start_us":775672,"threads_started":1,"wal-append.queue_time_us":790009}
I20260806 20:35:26.799883 16436 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.800130 14629 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
W20260806 20:35:26.781095 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 57286.7us
I20260806 20:35:26.814154 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1649) took 1149 ms. Trace:
I20260806 20:35:26.814666 14629 rpcz_store.cc:276] 0806 20:35:25.664521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.799549 (+1135028us) service_pool.cc:227] Handling call
0806 20:35:26.799921 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.813819 (+ 13898us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.814079 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:26.712373 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 204126us
I20260806 20:35:26.818143 14441 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:26.728289 14631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.009s sys 0.017s
I20260806 20:35:26.735805 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 487) took 1228 ms. Trace:
I20260806 20:35:26.789106 14595 leader_election.cc:398] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:26.793040 16558 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:26.793470 16545 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:26.794350 16542 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.794647 16474 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.840004 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:26.829447 16572 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:26.795353 14549 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 4. Ops: []
I20260806 20:35:26.547080 14552 rpcz_store.cc:276] 0806 20:35:25.028105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.636414 (+608309us) service_pool.cc:227] Handling call
0806 20:35:25.648279 (+ 11865us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.648491 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:26.023715 (+375224us) log.cc:842] Serialized 14 byte log entry
0806 20:35:26.534674 (+510959us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:26.534724 (+ 50us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:26.534765 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.534809 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:26.534934 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":883169,"raft.queue_time_us":108,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":8781,"spinlock_wait_cycles":78592,"thread_start_us":765614,"threads_started":2}
I20260806 20:35:26.807829 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1360) took 2859 ms. Trace:
I20260806 20:35:26.847357 14551 rpcz_store.cc:276] 0806 20:35:23.948847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.715451 (+766604us) service_pool.cc:227] Handling call
0806 20:35:25.184611 (+469160us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.217482 (+ 32871us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.807610 (+1590128us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x7b5400086bd0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:26.807768 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1078233984}
I20260806 20:35:26.808662 14624 raft_consensus.cc:1261] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:26.809262 16434 raft_consensus.cc:3032] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:26.768680 16461 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:26.738561 16556 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:26.788774 14630 rpcz_store.cc:276] 0806 20:35:25.663765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.730097 (+1066332us) service_pool.cc:227] Handling call
0806 20:35:26.739046 (+ 8949us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.740498 (+ 1452us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:26.740531 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:26.740579 (+ 48us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:26.740616 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.740681 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:26.740793 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:26.778095 16399 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:26.773589 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 287471us
I20260806 20:35:26.821820 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:26.780709 16280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54494 (request call id 817) took 4192 ms. Trace:
I20260806 20:35:26.824542 16533 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.790738 16122 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:26.821457 16525 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.881053 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:26.888780 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:26.889748 14630 raft_consensus.cc:2379] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:26.817991 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 301971us
I20260806 20:35:26.827148 14444 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.779740 15946 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:26.788698 16408 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.789017 16514 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.831269 14595 leader_election.cc:401] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 20:35:26.832927 14518 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.794911 16562 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:26.843079 16179 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:26.843983 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:26.874132 16455 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:26.903319 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 70485.5us
W20260806 20:35:26.828776 14631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.009s sys 0.017s
I20260806 20:35:26.831184 15186 rpcz_store.cc:276] 0806 20:35:25.502137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.512030 (+ 9893us) service_pool.cc:227] Handling call
0806 20:35:25.765285 (+253255us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:25.765510 (+ 225us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:25.765732 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:35:26.613527 (+847795us) log.cc:842] Serialized 14 byte log entry
0806 20:35:26.729335 (+115808us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:26.729417 (+ 82us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:26.729479 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.730739 (+ 1260us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:26.730906 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9066,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":175,"spinlock_wait_cycles":10315520,"thread_start_us":106309,"threads_started":1,"wal-append.queue_time_us":186908}
I20260806 20:35:26.865029 16574 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:26.856045 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:26.826591 16436 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.862527 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:26.829118 16473 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.858255 16572 raft_consensus.cc:686] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:26.858716 16556 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:26.853066 16429 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.903581 16303 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.878199 16566 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.928687 16303 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.929013 16566 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.855731 16434 raft_consensus.cc:729] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:26.930310 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:26.930222 16434 consensus_queue.cc:260] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:26.931206 15180 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 350) took 1754 ms (client timeout 2097 ms). Trace:
W20260806 20:35:26.931730 15180 rpcz_store.cc:269] 0806 20:35:25.176756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.217743 (+ 40987us) service_pool.cc:227] Handling call
0806 20:35:26.930219 (+1712476us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593c6c 00007fb9b0593bd6 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:26.931142 (+ 923us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54526208}
I20260806 20:35:26.934087 16292 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:26.880564 16525 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.919265 16569 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:26.943801 16545 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.917809 16408 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:26.944891 16408 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.946094 16545 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.893134 14625 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:26.900782 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:26.952590 14630 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:26.891877 14628 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:26.903405 16455 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.842346 16558 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.869268 16280 rpcz_store.cc:276] 0806 20:35:22.543756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:22.543914 (+ 158us) service_pool.cc:227] Handling call
0806 20:35:22.620042 (+ 76128us) log.cc:842] Serialized 508 byte log entry
0806 20:35:24.803182 (+2183140us) spinlock_profiling.cc:243] Waited 56.6 ms on lock 0x7b5400058d20. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b3481771 00007fb9b348c220 00007fb9b348c1c8 00007fb9b348c148 00007fb9b348c103 00007fb9b348b38c 00007fb9b94fc206 00007fb9b94f4808 00007fb9b34c838b 00007fb9b34cae80
0806 20:35:26.735682 (+1932500us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15810,"raft.queue_time_us":136378,"raft.run_cpu_time_us":1976,"raft.run_wall_time_us":47985,"spinlock_wait_cycles":130126592,"thread_start_us":401323,"threads_started":2,"wal-append.queue_time_us":399559,"wal-append.run_cpu_time_us":23868,"wal-append.run_wall_time_us":2797203}
I20260806 20:35:26.960877 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:26.961912 14630 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:26.939479 16572 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.962208 16558 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.963054 16309 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.964020 15153 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:26.904995 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 77909.1us
I20260806 20:35:26.915273 14518 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:26.912523 14479 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:26.884357 16424 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.880060 16419 raft_consensus.cc:3032] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:26.889163 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1652) took 1220 ms. Trace:
I20260806 20:35:26.922072 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:26.904050 16533 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:26.923504 16556 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:26.919106 16436 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:26.924124 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 135158us
W20260806 20:35:26.924218 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 220) took 2273 ms (client timeout 2404 ms). Trace:
I20260806 20:35:26.857769 16157 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.985037 14595 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:26.985982 16436 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.986177 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:26.986505 16157 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.944254 16569 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.987139 14477 raft_consensus.cc:1226] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:26.866561 16461 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.987784 16546 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:26.892827 16456 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:26.988484 16569 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:26.989025 16397 consensus_meta.cc:287] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.501s user 0.005s sys 0.005s
I20260806 20:35:26.989362 16461 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.989557 14444 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.931277 16434 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:26.921384 16473 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.947719 16408 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.948426 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1544) took 1431 ms. Trace:
I20260806 20:35:27.000404 14444 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.000571 14553 rpcz_store.cc:276] 0806 20:35:25.517037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.809576 (+292539us) service_pool.cc:227] Handling call
0806 20:35:25.854923 (+ 45347us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.601222 (+746299us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.948205 (+346983us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54000b40d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:26.948371 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1714604416}
I20260806 20:35:26.943683 16555 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:26.950217 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 222) took 2298 ms (client timeout 2442 ms). Trace:
I20260806 20:35:26.846590 16286 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260806 20:35:26.953056 14521 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:27.004060 14474 rpcz_store.cc:269] 0806 20:35:24.651348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.657875 (+ 6527us) service_pool.cc:227] Handling call
0806 20:35:26.950004 (+2292129us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b5400227254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:26.950146 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4449538304}
W20260806 20:35:27.014086 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 61140us
I20260806 20:35:26.980576 16558 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:26.955235 14629 raft_consensus.cc:1226] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
W20260806 20:35:26.966199 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 51009us
I20260806 20:35:26.967000 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.015475 14521 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:27.016932 14479 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:27.017427 14518 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.017624 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.017757 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:26.878682 16122 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.018543 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:27.021020 16447 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.873023 16410 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:26.904294 16562 raft_consensus.cc:686] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:27.018406 14479 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:26.975802 14470 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:26.977636 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.029203 14625 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:26.982544 16361 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:26.987047 14595 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:26.919281 16166 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:35:26.984735 14475 rpcz_store.cc:269] 0806 20:35:24.650673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.658237 (+ 7564us) service_pool.cc:227] Handling call
0806 20:35:26.924013 (+2265776us) spinlock_profiling.cc:243] Waited 9.6 ms on lock 0x7b54001a63d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:26.924137 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22081920}
I20260806 20:35:26.949429 16285 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:26.995668 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 129047us
I20260806 20:35:26.996547 16262 raft_consensus.cc:3032] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:27.060624 16262 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:26.993630 16031 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260806 20:35:27.067802 14518 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:26.975225 16419 raft_consensus.cc:729] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:27.070153 16419 consensus_queue.cc:260] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.071782 16419 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:27.072600 16578 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.073653 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:27.003854 16263 consensus_queue.cc:1046] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 20:35:26.997648 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1377) took 2137 ms. Trace:
I20260806 20:35:27.075893 14546 rpcz_store.cc:276] 0806 20:35:24.860485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.563112 (+702627us) service_pool.cc:227] Handling call
0806 20:35:25.629876 (+ 66764us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.504952 (+875076us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.997407 (+492455us) spinlock_profiling.cc:243] Waited 61.1 ms on lock 0x7b54002db4d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:26.997561 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":140543360}
I20260806 20:35:26.977288 16179 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260806 20:35:26.998423 16550 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:27.077613 16550 raft_consensus.cc:686] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:26.989410 16397 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:27.012744 16555 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:26.954296 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.004851 14444 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:27.079610 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 74835.2us
I20260806 20:35:26.990521 16456 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.017922 16533 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:27.004733 16455 consensus_queue.cc:1046] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260806 20:35:27.086416 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:26.877921 16494 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.086719 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.023188 16447 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.088390 16494 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.018448 14474 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:27.089387 16122 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:27.086470 16410 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.086757 16533 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:27.090564 16533 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:27.090885 16473 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:26.999541 16157 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:27.086478 16555 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:27.092903 16555 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:27.093258 16555 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.095362 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:27.096017 14546 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:27.037667 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.020288 16436 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:27.039906 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.046298 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 236) took 1619 ms. Trace:
I20260806 20:35:27.047407 14626 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260806 20:35:27.049510 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 62541.6us
I20260806 20:35:26.975425 14624 rpcz_store.cc:276] 0806 20:35:25.665384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.807838 (+1142454us) service_pool.cc:227] Handling call
0806 20:35:26.808230 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.856433 (+ 48203us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.884443 (+ 28010us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:26.921574 16574 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.021253 16461 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:27.079054 14628 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:26.988406 16546 raft_consensus.cc:2741] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:35:27.078554 16550 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.073292 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:27.088485 14469 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:27.044994 16562 consensus_queue.cc:237] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.073276 16578 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.024894 16408 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:27.013487 16399 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.134740 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.132982 16436 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:27.139505 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1316) took 2125 ms. Trace:
I20260806 20:35:27.141287 14479 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:27.035632 16402 raft_consensus.cc:686] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:27.139631 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 53417.8us
I20260806 20:35:27.050208 16566 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:27.134697 14470 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:27.145689 14553 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:27.243398 14444 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:27.135049 14632 rpcz_store.cc:276] 0806 20:35:25.426554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.585420 (+158866us) service_pool.cc:227] Handling call
0806 20:35:25.621052 (+ 35632us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:25.621437 (+ 385us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:25.621465 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:25.646629 (+ 25164us) log.cc:842] Serialized 26 byte log entry
0806 20:35:26.877693 (+1231064us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b1800300a00. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b0598f96 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39
0806 20:35:26.879596 (+ 1903us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b1800300a00. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a411 00007fb9b34c75cf
0806 20:35:26.974600 (+ 95004us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:26.974769 (+ 169us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:26.974841 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:27.046048 (+ 71207us) raft_consensus.cc:1671] finished
0806 20:35:27.046090 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:27.046228 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11436,"raft.queue_time_us":2628,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":47908,"spinlock_wait_cycles":104403072,"thread_start_us":47712,"threads_started":1,"wal-append.queue_time_us":1360301}
I20260806 20:35:27.169334 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:27.167425 14551 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:27.096335 15946 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:27.150430 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.175391 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.246029 15151 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:27.080155 16545 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:27.173230 14550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 248) took 4012 ms (client timeout 1617 ms). Trace:
W20260806 20:35:27.255745 14550 rpcz_store.cc:269] 0806 20:35:23.160869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:23.178289 (+ 17420us) service_pool.cc:227] Handling call
0806 20:35:27.172924 (+3994635us) spinlock_profiling.cc:243] Waited 3.12 s on lock 0x7b54002d9bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:27.173134 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7170038144}
I20260806 20:35:27.235198 14626 rpcz_store.cc:276] 0806 20:35:25.007952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.279081 (+271129us) service_pool.cc:227] Handling call
0806 20:35:27.028294 (+1749213us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:27.028559 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:27.132963 (+104404us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:27.133295 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6667648}
I20260806 20:35:27.143637 16286 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:27.250425 14630 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.150755 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:27.252868 14595 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:27.254323 14476 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:27.259589 14473 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
W20260806 20:35:27.260106 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 253475us
I20260806 20:35:27.100965 16303 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.136s
I20260806 20:35:27.173861 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.231844 15821 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.143848 16587 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:27.170535 16456 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.173032 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.173460 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.242211 16436 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:27.168048 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.232167 16410 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:27.167023 16309 raft_consensus.cc:3032] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:27.216249 16546 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:27.248835 16166 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.363623 16166 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.364326 16309 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:27.363734 16556 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.365943 16309 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.367834 16309 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.367312 16546 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.266557 16286 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.168859 16473 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.112s
I20260806 20:35:27.195915 16408 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:27.248911 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.248646 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.259912 16545 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:27.168324 16574 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:27.260135 16572 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:27.331410 14355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b16d0272c85445ed95a93660f8851656 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:35:27.337661 16396 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:27.245214 16494 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:27.317507 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.343247 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.331723 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:27.345132 16411 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:27.347183 16399 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.295332 16292 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.350070 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.352159 14628 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:27.281431 16357 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:27.357033 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 104291us
I20260806 20:35:27.281663 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.291679 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:27.373783 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 193920us
I20260806 20:35:27.292270 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 234) took 1855 ms. Trace:
I20260806 20:35:27.292665 16525 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.386260 14630 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:27.349735 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.393054 14521 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:27.390225 14626 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:27.409407 14632 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:27.362361 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.409423 14470 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:27.422524 14518 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.355594 16456 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:27.412681 14355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 118206us
I20260806 20:35:27.415380 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.373616 16408 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.419394 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.420732 14521 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:27.449800 14521 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:27.381449 16545 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:27.409811 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.451531 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.423132 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.423925 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.410269 16292 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.412828 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.424818 15183 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:27.425127 16434 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:27.472625 16434 raft_consensus.cc:686] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:27.473695 16434 consensus_queue.cc:237] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.455572 16473 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:27.474825 16473 raft_consensus.cc:686] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:27.425707 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.475704 16473 consensus_queue.cc:237] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.476846 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.477936 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.478367 16465 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:27.478513 14632 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:27.479488 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.480888 14475 raft_consensus.cc:2379] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:27.481161 15153 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.481819 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.482189 16225 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:27.482649 14475 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:27.483532 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.483971 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.484138 14475 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:27.427649 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.487005 15181 raft_consensus.cc:2379] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:27.487805 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.446233 16399 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 20:35:27.488914 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.490490 15181 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:27.428757 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:27.429733 16586 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260806 20:35:27.317075 15821 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.372732 16410 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.495013 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.495841 14478 raft_consensus.cc:2379] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:27.496500 16546 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:27.496672 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.496822 16410 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.497517 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:27.498256 14550 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:27.499225 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.499955 14475 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:27.500917 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.418867 16574 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.502007 16574 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.435029 14551 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:27.437397 16122 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:27.437536 16384 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.447155 16402 consensus_queue.cc:237] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.504788 16122 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.504890 16384 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.442279 14546 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:27.443886 16411 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:27.379786 16572 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:27.452209 14628 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.452942 14470 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:27.424067 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:27.412662 14624 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:27.462178 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.463784 15182 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:27.464733 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:27.466329 16292 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.471843 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.480755 16578 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.392655 16396 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:27.458642 16587 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:27.485252 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:27.425848 16286 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:27.501257 15821 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:27.502635 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.437119 16542 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:27.422433 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:27.450800 16525 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:27.508737 14475 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:27.509300 14444 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:27.393833 16494 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.423496 16357 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.511864 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.514307 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.824574 14630 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:27.697046 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:35:27.696101 14555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 257) took 2646 ms (client timeout 1854 ms). Trace:
W20260806 20:35:27.746743 16558 consensus_meta.cc:287] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.719s user 0.011s sys 0.000s
I20260806 20:35:27.827912 14552 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
W20260806 20:35:27.830287 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 406273us
I20260806 20:35:27.854868 14518 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.856808 14518 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.852325 14632 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:27.829974 14595 connection.cc:232] RPC call timeout handler was delayed by -1.104731773986714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1506s for a response
W20260806 20:35:27.859489 14595 connection.cc:232] RPC call timeout handler was delayed by -1.0534317739948165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
W20260806 20:35:27.828655 14555 rpcz_store.cc:269] 0806 20:35:25.050043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.313778 (+263735us) service_pool.cc:227] Handling call
0806 20:35:27.695880 (+2382102us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000b36d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:27.696020 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2597438720}
I20260806 20:35:27.827633 14626 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:35:27.847901 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 145678us
I20260806 20:35:27.866366 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.461361 16556 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:27.504364 16574 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:27.825337 16533 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.734s user 0.012s sys 0.000s
I20260806 20:35:27.880780 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.871873 14595 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:27.808630 16563 log.cc:924] Time spent T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
W20260806 20:35:27.685436 16419 consensus_meta.cc:287] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.597s user 0.012s sys 0.001s
I20260806 20:35:27.873613 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:27.883059 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 373842us
I20260806 20:35:27.684511 16122 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:27.883947 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.883267 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.891533 14475 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:27.844103 16558 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:27.849143 16572 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.870760 16473 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:27.864140 16492 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:27.893256 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.895115 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:27.850307 16396 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.607052 16545 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:27.910193 16384 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:27.912242 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.912865 14625 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:27.913290 14624 raft_consensus.cc:2396] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:27.914243 16494 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:27.915683 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.592957 16411 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.875506 16525 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 20:35:27.886031 16555 consensus_meta.cc:287] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.792s user 0.007s sys 0.004s
I20260806 20:35:27.903164 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:27.903276 14547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1375) took 3083 ms (client timeout 3313 ms). Trace:
W20260806 20:35:27.923437 14547 rpcz_store.cc:269] 0806 20:35:24.821922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.470662 (+648740us) service_pool.cc:227] Handling call
0806 20:35:27.903096 (+2432434us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b54000b45d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:27.903214 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148595456}
I20260806 20:35:27.846992 16415 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:27.917764 14518 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.916736 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:27.925828 14632 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:27.831990 16587 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:27.918462 14475 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:27.890543 15821 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:27.865037 16542 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:27.933204 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.969339 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1593) took 1107 ms. Trace:
I20260806 20:35:27.969529 14626 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:27.926724 14628 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:27.969746 14473 rpcz_store.cc:276] 0806 20:35:26.861617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.861809 (+ 192us) service_pool.cc:227] Handling call
0806 20:35:26.862268 (+ 459us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:27.259991 (+397723us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:27.969073 (+709082us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54003f4dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:27.969278 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":913337216}
I20260806 20:35:27.970957 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.971468 14628 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:27.971289 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.967106 15150 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:27.973744 15153 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:27.918027 15179 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:27.902774 16473 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:27.976958 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.975211 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:27.990211 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 128436us
I20260806 20:35:27.998377 14470 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:27.967005 14518 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:27.929858 15821 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:27.895426 16574 consensus_queue.cc:1046] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260806 20:35:27.888545 16533 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.001156 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.003188 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.924041 16545 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:27.974552 15178 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:35:28.004287 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:27.968905 16122 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 20:35:28.001693 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 73229.4us
I20260806 20:35:28.003460 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:27.926177 16456 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.006187 14630 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
W20260806 20:35:27.976578 16309 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.604s user 0.013s sys 0.003s
W20260806 20:35:28.014629 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 85846.4us
I20260806 20:35:27.969444 16415 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:28.011876 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.013417 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.022409 14475 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:27.939662 16587 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:28.017866 14595 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.014596 14473 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:28.018671 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.000677 16473 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:28.025347 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.025892 14473 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:28.026635 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.027846 14470 raft_consensus.cc:2379] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:28.030905 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.032883 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.033150 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.007478 15821 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.002851 16434 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:28.033710 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.027460 16292 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:28.039376 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.044742 14473 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:28.034093 15150 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:28.036687 16398 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.040s sys 0.070s
I20260806 20:35:28.041458 14475 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:27.935431 16555 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.037348 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.054958 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:28.055469 16555 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.055673 14624 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:28.056380 14474 raft_consensus.cc:2379] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:28.040223 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:28.058279 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.059355 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.059522 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.060896 16286 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:28.061411 16357 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.061709 16415 raft_consensus.cc:686] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:28.062912 16542 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:28.062871 16415 consensus_queue.cc:237] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.036736 16217 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:28.064728 16217 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:28.065116 16217 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:28.066370 16574 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:28.066713 16166 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.068383 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:28.068632 16292 raft_consensus.cc:686] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:28.047016 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.050323 16533 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.069403 16524 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.069954 14470 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:28.069701 16292 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.070924 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.071673 14470 raft_consensus.cc:2379] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:28.073128 16574 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:28.073724 16574 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.075753 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1591) took 1214 ms. Trace:
I20260806 20:35:28.075842 16574 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.076280 14477 rpcz_store.cc:276] 0806 20:35:26.860879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.863200 (+ 2321us) service_pool.cc:227] Handling call
0806 20:35:26.872253 (+ 9053us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.987540 (+115287us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.075502 (+1087962us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540017e154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:28.075685 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":263736064}
I20260806 20:35:28.077574 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.078904 14477 raft_consensus.cc:1226] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:28.079430 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1600) took 1156 ms. Trace:
I20260806 20:35:28.072384 16434 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:28.069123 14624 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:28.080523 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.081354 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1548) took 1570 ms. Trace:
I20260806 20:35:28.083190 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:28.048076 16398 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.565s user 0.040s sys 0.070s
I20260806 20:35:28.079902 14518 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:28.103859 14548 rpcz_store.cc:276] 0806 20:35:26.510691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.080197 (+1569506us) service_pool.cc:227] Handling call
0806 20:35:28.081279 (+ 1082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.104207 14628 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:35:28.042096 16285 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.987s user 0.010s sys 0.002s
I20260806 20:35:28.105048 16285 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:28.093477 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.105198 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.108563 16396 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.110841 16396 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:28.111380 16542 consensus_queue.cc:1046] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:28.046561 16525 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:28.095714 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 96818us
I20260806 20:35:28.115257 14595 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.114775 15153 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:28.117538 14518 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:28.117991 14546 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:28.118698 14518 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.123302 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.124650 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.103058 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.125458 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:28.119884 14518 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:28.128703 14478 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:28.096887 15150 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:28.131524 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 81766.3us
I20260806 20:35:28.079730 14477 rpcz_store.cc:276] 0806 20:35:26.923106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.078426 (+1155320us) service_pool.cc:227] Handling call
0806 20:35:28.078747 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.079228 (+ 481us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.079386 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.085695 16122 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:35:28.142257 16600 log.cc:924] Time spent T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20260806 20:35:28.129608 16357 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:28.142578 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.134289 16396 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:28.079921 16566 consensus_meta.cc:287] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.833s user 0.003s sys 0.004s
I20260806 20:35:28.143848 16396 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:28.144253 16396 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:28.144122 16566 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:28.135923 14475 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:28.136049 16574 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:28.146435 16574 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:28.146807 16574 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:28.146881 14477 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:28.135918 16562 consensus_queue.cc:1046] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260806 20:35:28.152479 16562 raft_consensus.cc:3032] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:28.136157 16533 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:28.152802 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:28.152832 16562 raft_consensus.cc:729] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:28.154430 16562 consensus_queue.cc:260] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.155378 14550 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:28.155638 16562 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:28.156389 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.156977 14550 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.158967 14595 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.141337 16286 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:28.162994 14595 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.147351 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.164371 16384 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:28.129891 16545 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.120136 16157 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 20:35:28.136135 16524 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:28.165477 16524 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:28.165764 14595 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:28.129739 16542 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:28.166711 16384 raft_consensus.cc:2741] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:35:28.166672 16542 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.169265 16411 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:28.169669 16542 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:28.166009 16524 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.174060 14630 raft_consensus.cc:1226] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:28.174710 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1658) took 1593 ms. Trace:
I20260806 20:35:28.175014 14630 rpcz_store.cc:276] 0806 20:35:26.581444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.173546 (+1592102us) service_pool.cc:227] Handling call
0806 20:35:28.173871 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.174477 (+ 606us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.174658 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.163656 16122 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:28.176992 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:28.177564 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1372) took 3433 ms. Trace:
I20260806 20:35:28.177690 15183 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:28.177939 14549 rpcz_store.cc:276] 0806 20:35:24.744666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.477681 (+733015us) service_pool.cc:227] Handling call
0806 20:35:25.497335 (+ 19654us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:26.846624 (+1349289us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.177348 (+1330724us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54000b3bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:28.177508 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2984616832}
I20260806 20:35:28.179257 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.179962 14549 raft_consensus.cc:2379] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:28.160826 16217 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.185271 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.185973 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.207000 14553 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.186919 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1661) took 1366 ms. Trace:
I20260806 20:35:28.187611 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:27.411543 14631 rpcz_store.cc:276] 0806 20:35:25.425990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.507717 (+ 81727us) service_pool.cc:227] Handling call
0806 20:35:25.569168 (+ 61451us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:25.862872 (+293704us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:25.925795 (+ 62923us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:26.068751 (+142956us) log.cc:842] Serialized 26 byte log entry
0806 20:35:26.921691 (+852940us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:26.939797 (+ 18106us) log.cc:842] Serialized 14 byte log entry
0806 20:35:26.940015 (+ 218us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:27.175956 (+235941us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54003bf954. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:27.176011 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:27.176066 (+ 55us) raft_consensus.cc:1671] finished
0806 20:35:27.254155 (+ 78089us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:27.281356 (+ 27201us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1449,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":110,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":553081728,"thread_start_us":758134,"threads_started":1,"wal-append.queue_time_us":615870}
I20260806 20:35:28.149524 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.215263 14547 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:28.142028 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:28.119292 16436 consensus_meta.cc:287] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.770s user 0.009s sys 0.002s
I20260806 20:35:28.193506 14633 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:28.198511 16574 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.202762 14550 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:28.112315 16285 raft_consensus.cc:3032] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:28.183887 16514 raft_consensus.cc:3032] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 20:35:28.207893 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 84924.4us
I20260806 20:35:28.207751 14630 rpcz_store.cc:276] 0806 20:35:26.820167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.175885 (+1355718us) service_pool.cc:227] Handling call
0806 20:35:28.176196 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.176364 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.176390 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.176433 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:28.186186 (+ 9753us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.186267 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.186394 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.221325 14475 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:28.213893 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 79638.1us
I20260806 20:35:28.214738 14444 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:28.211118 14632 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:28.226284 14474 raft_consensus.cc:1226] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:28.228190 14474 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:28.229871 14474 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:28.234392 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:28.235955 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.237470 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.253422 14444 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:28.255769 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.262117 14632 raft_consensus.cc:1226] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:35:28.190764 16593 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:28.101408 16434 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.263180 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1666) took 1416 ms. Trace:
I20260806 20:35:28.275182 14632 rpcz_store.cc:276] 0806 20:35:26.847111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.240600 (+1393489us) service_pool.cc:227] Handling call
0806 20:35:28.241032 (+ 432us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:28.262769 (+ 21737us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.263093 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.141610 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:28.252522 15946 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:28.270498 15821 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.271294 16542 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:28.220037 16514 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:28.123580 16309 consensus_queue.cc:1046] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.106s
I20260806 20:35:28.219843 16285 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:35:28.282414 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 90923.2us
I20260806 20:35:28.134423 16525 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.296684 16456 consensus_queue.cc:1046] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:35:28.302110 16514 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.281051 15946 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:28.304761 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 51449.3us
I20260806 20:35:28.282250 14624 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:28.306988 15187 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:28.225100 16436 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:28.307780 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1587) took 1476 ms (client timeout 1635 ms). Trace:
I20260806 20:35:28.307945 14475 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:28.308738 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 247) took 1317 ms (client timeout 1602 ms). Trace:
W20260806 20:35:28.354789 14625 rpcz_store.cc:269] 0806 20:35:26.991576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.050934 (+ 59358us) service_pool.cc:227] Handling call
0806 20:35:28.308531 (+1257597us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b5400051254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.308669 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1600678144}
I20260806 20:35:28.267313 16399 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:28.311787 14546 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:28.320535 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 246) took 1433 ms. Trace:
I20260806 20:35:28.358606 14632 rpcz_store.cc:276] 0806 20:35:26.887055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.277319 (+1390264us) service_pool.cc:227] Handling call
0806 20:35:28.294409 (+ 17090us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.294635 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.294673 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.294731 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:28.320147 (+ 25416us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.320253 (+ 106us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.320418 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.322407 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.322736 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:28.275802 16593 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:28.293304 16408 consensus_meta.cc:287] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.823s user 0.010s sys 0.001s
I20260806 20:35:28.382169 14549 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:28.389669 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 67014.1us
I20260806 20:35:28.394192 14625 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:28.395519 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1668) took 1459 ms. Trace:
I20260806 20:35:28.396194 14625 rpcz_store.cc:276] 0806 20:35:26.935928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.356118 (+1420190us) service_pool.cc:227] Handling call
0806 20:35:28.393927 (+ 37809us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.394983 (+ 1056us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.395393 (+ 410us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.399904 14554 raft_consensus.cc:1226] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:28.330742 15946 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.403419 14444 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:28.397440 14473 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
W20260806 20:35:28.426971 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 133166us
I20260806 20:35:28.430847 14595 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.431541 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 235) took 1077 ms. Trace:
I20260806 20:35:28.431864 14473 rpcz_store.cc:276] 0806 20:35:27.353829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.038998 (+685169us) service_pool.cc:227] Handling call
0806 20:35:28.431481 (+392483us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.432457 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.433298 14595 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.434020 15150 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:28.435096 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:28.406423 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1672) took 1366 ms. Trace:
W20260806 20:35:28.410588 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 48960.9us
I20260806 20:35:28.435377 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.446688 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 248) took 1402 ms. Trace:
I20260806 20:35:28.368362 16399 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:28.455000 14625 rpcz_store.cc:276] 0806 20:35:27.043958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.405554 (+1361596us) service_pool.cc:227] Handling call
0806 20:35:28.446057 (+ 40503us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.446288 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.446323 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.446375 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:28.446434 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.446500 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.446619 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.375017 16593 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:28.431944 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 192138us
I20260806 20:35:28.463946 14473 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:28.467315 14548 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:28.436165 14632 rpcz_store.cc:276] 0806 20:35:27.039821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.360638 (+1320817us) service_pool.cc:227] Handling call
0806 20:35:28.405798 (+ 45160us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.406033 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.406070 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.406124 (+ 54us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:28.406163 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.406226 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.406354 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:28.470475 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1651) took 2805 ms (client timeout 3226 ms). Trace:
W20260806 20:35:28.486408 14627 rpcz_store.cc:269] 0806 20:35:25.665067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.530514 (+865447us) service_pool.cc:227] Handling call
0806 20:35:28.470300 (+1939786us) spinlock_profiling.cc:243] Waited 32.4 ms on lock 0x7b540009f454. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.470420 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74459008}
I20260806 20:35:28.487138 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.468227 14518 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:28.375213 16555 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20260806 20:35:28.368527 16396 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:28.462417 16492 consensus_meta.cc:287] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.543s user 0.009s sys 0.000s
I20260806 20:35:28.454474 16122 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:28.401537 16408 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.461650 16399 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:28.458875 16157 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:28.514693 14550 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 20:35:28.511652 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 43494.3us
I20260806 20:35:28.425804 16514 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:28.425302 16602 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:28.426054 16524 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.518877 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.491155 16384 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:28.463228 16456 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:28.539588 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 245) took 1705 ms. Trace:
I20260806 20:35:28.540266 14631 rpcz_store.cc:276] 0806 20:35:26.833955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.228825 (+1394870us) service_pool.cc:227] Handling call
0806 20:35:28.273683 (+ 44858us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.273961 (+ 278us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:28.455465 (+181504us) log.cc:842] Serialized 14 byte log entry
0806 20:35:28.539214 (+ 83749us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.539265 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:28.539317 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.539406 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.539527 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":264204,"threads_started":2}
W20260806 20:35:28.368093 14469 rpcz_store.cc:269] 0806 20:35:26.831505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.832519 (+ 1014us) service_pool.cc:227] Handling call
0806 20:35:28.307530 (+1475011us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54003c4954. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.307708 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257974528}
I20260806 20:35:28.513105 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.526801 14518 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:28.549284 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.433811 16309 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:28.550261 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 74528.3us
I20260806 20:35:28.456187 16542 consensus_queue.cc:1035] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260806 20:35:28.462033 16411 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:28.558317 14548 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:28.504760 14547 raft_consensus.cc:2932] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:28.518044 16415 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:28.524068 16303 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:28.533257 14632 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:28.568222 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1330) took 1302 ms. Trace:
I20260806 20:35:28.568904 14550 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 20:35:28.551575 16572 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.633s user 0.010s sys 0.000s
I20260806 20:35:28.526074 16492 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:28.568720 14632 rpcz_store.cc:276] 0806 20:35:27.265546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.532102 (+1266556us) service_pool.cc:227] Handling call
0806 20:35:28.533014 (+ 912us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.567892 (+ 34878us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.568152 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.551609 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
W20260806 20:35:28.570530 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 110503us
I20260806 20:35:28.570894 14631 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:28.518030 16286 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260806 20:35:28.608649 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1337) took 1254 ms. Trace:
I20260806 20:35:28.552145 16309 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:28.611855 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 164677us
W20260806 20:35:28.611963 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 60433.8us
I20260806 20:35:28.611506 14631 rpcz_store.cc:276] 0806 20:35:27.354592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.542099 (+1187507us) service_pool.cc:227] Handling call
0806 20:35:28.570665 (+ 28566us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.608349 (+ 37684us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.608589 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.432704 16285 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.618237 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:28.621721 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.565758 16494 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:28.562117 16415 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:28.577250 16473 consensus_meta.cc:287] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.529s user 0.005s sys 0.005s
I20260806 20:35:28.636608 14550 raft_consensus.cc:2379] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:28.565930 16411 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.566923 16574 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.637761 14632 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:28.568635 16456 raft_consensus.cc:686] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:28.622623 16309 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.658758 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1339) took 1289 ms. Trace:
I20260806 20:35:28.656929 16545 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:28.657021 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.607475 15821 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.651080 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.664377 14553 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:28.661006 14632 rpcz_store.cc:276] 0806 20:35:27.369756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.609706 (+1239950us) service_pool.cc:227] Handling call
0806 20:35:28.637576 (+ 27870us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.658124 (+ 20548us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.658619 (+ 495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:28.644325 16415 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.660324 14550 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:28.624595 16434 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:28.677529 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 83469us
I20260806 20:35:28.677927 14550 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:28.663031 14625 raft_consensus.cc:2379] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:28.639853 16494 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:28.645823 16217 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:28.664186 16604 log.cc:924] Time spent T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20260806 20:35:28.645233 16525 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.607842 16572 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.655853 16436 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.673687 14469 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:28.673887 16122 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:28.646754 16473 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.690663 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.697789 16456 consensus_queue.cc:237] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.675027 16555 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:28.677614 16545 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:28.703182 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:28.737756 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 119610us
I20260806 20:35:28.681565 16494 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:28.689144 16622 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:28.689880 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:28.700779 16122 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:28.753978 14553 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
W20260806 20:35:28.746443 16606 log.cc:924] Time spent T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20260806 20:35:28.741899 16396 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:28.760965 16607 log.cc:924] Time spent T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
W20260806 20:35:28.761742 16616 log.cc:924] Time spent T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
W20260806 20:35:28.762447 16617 log.cc:924] Time spent T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20260806 20:35:28.763060 16614 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
W20260806 20:35:28.763634 16610 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260806 20:35:28.750337 16622 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:28.764292 14442 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.785s (SENT) [suppressed 17 similar messages]
I20260806 20:35:28.766136 16436 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:28.748814 16524 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:28.810006 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 7f41753baced49a1997940dfc28b2e00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:28.781558 14549 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:28.747212 16408 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:28.773041 16611 log.cc:924] Time spent T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260806 20:35:28.863253 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 249) took 1769 ms. Trace:
W20260806 20:35:28.794301 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 182676us
I20260806 20:35:28.740217 16555 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.863871 14627 rpcz_store.cc:276] 0806 20:35:27.093476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.487847 (+1394371us) service_pool.cc:227] Handling call
0806 20:35:28.520315 (+ 32468us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:28.646295 (+125980us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.646336 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:28.664199 (+ 17863us) log.cc:842] Serialized 26 byte log entry
0806 20:35:28.738983 (+ 74784us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:28.739038 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.739127 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:28.862983 (+123856us) raft_consensus.cc:1671] finished
0806 20:35:28.863006 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.863180 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":127710,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":2694,"thread_start_us":158781,"threads_started":2,"wal-append.queue_time_us":130865}
I20260806 20:35:28.866173 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.866601 16555 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.741423 16545 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.755966 16122 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.806418 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.869365 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.870604 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.815013 16625 consensus_queue.cc:1046] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.072s
I20260806 20:35:28.824084 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1656) took 2343 ms. Trace:
W20260806 20:35:28.817471 16615 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
W20260806 20:35:28.818343 16609 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
W20260806 20:35:28.816346 16591 log.cc:924] Time spent T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.405s user 0.000s sys 0.012s
I20260806 20:35:28.820781 16460 raft_consensus.cc:3032] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:28.821688 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 246) took 1126 ms. Trace:
I20260806 20:35:28.824420 16399 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.826615 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.830010 16309 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:28.832207 14551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 269) took 1765 ms (client timeout 2230 ms). Trace:
I20260806 20:35:28.833012 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 240) took 3166 ms. Trace:
W20260806 20:35:28.894397 14551 rpcz_store.cc:269] 0806 20:35:27.066924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.073372 (+ 6448us) service_pool.cc:227] Handling call
0806 20:35:28.832033 (+1758661us) spinlock_profiling.cc:243] Waited 93 ms on lock 0x7b54000a8f54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.832148 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":213786368}
I20260806 20:35:28.894517 14629 rpcz_store.cc:276] 0806 20:35:25.667005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.815911 (+1148906us) service_pool.cc:227] Handling call
0806 20:35:26.830803 (+ 14892us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:27.026332 (+195529us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.832772 (+1806440us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400302d54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:28.832935 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":285608704}
W20260806 20:35:28.834110 14633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 232) took 3408 ms (client timeout 1854 ms). Trace:
W20260806 20:35:28.837016 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 98302us
I20260806 20:35:28.895982 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.896565 14595 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:28.837311 16547 consensus_queue.cc:1035] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:28.842549 16473 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.811826 16436 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.830019 16280 raft_consensus.cc:3032] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:28.851366 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 228) took 1619 ms. Trace:
I20260806 20:35:28.924372 14470 rpcz_store.cc:276] 0806 20:35:27.232357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.101122 (+868765us) service_pool.cc:227] Handling call
0806 20:35:28.123349 (+ 22227us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:28.135415 (+ 12066us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.135461 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:28.141649 (+ 6188us) log.cc:842] Serialized 26 byte log entry
0806 20:35:28.425798 (+284149us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:28.425900 (+ 102us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.426159 (+ 259us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:28.851138 (+424979us) raft_consensus.cc:1671] finished
0806 20:35:28.851165 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.851307 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":67599,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":16361,"thread_start_us":283234,"threads_started":1,"wal-append.queue_time_us":416428}
I20260806 20:35:28.853282 16411 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.854223 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 230) took 1594 ms. Trace:
W20260806 20:35:28.855535 16587 consensus_meta.cc:287] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.806s user 0.009s sys 0.001s
I20260806 20:35:28.928716 16587 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.929700 14469 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:28.931228 16587 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.862532 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:28.764430 16618 log.cc:924] Time spent T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
W20260806 20:35:28.933099 16608 log.cc:924] Time spent T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.276s user 0.000s sys 0.000s
I20260806 20:35:28.928232 14477 rpcz_store.cc:276] 0806 20:35:27.259624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.151553 (+891929us) service_pool.cc:227] Handling call
0806 20:35:28.151994 (+ 441us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:28.214625 (+ 62631us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.214660 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:28.253586 (+ 38926us) log.cc:842] Serialized 26 byte log entry
0806 20:35:28.427085 (+173499us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:28.427128 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.427202 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:28.853957 (+426755us) raft_consensus.cc:1671] finished
0806 20:35:28.853980 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.854154 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53773,"raft.queue_time_us":67350,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":10880,"thread_start_us":139031,"threads_started":1,"wal-append.queue_time_us":388667}
I20260806 20:35:28.935583 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:28.936338 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.936962 14477 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:28.937806 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:28.910033 16514 consensus_queue.cc:1046] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260806 20:35:28.938776 14551 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:28.940032 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:28.882736 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 230012us
I20260806 20:35:28.924247 16280 raft_consensus.cc:729] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:28.942001 16572 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.942790 16634 consensus_queue.cc:1046] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:28.943161 16621 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:28.944372 16634 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:28.945086 16634 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:28.946456 14478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1464) took 1519 ms (client timeout 1923 ms). Trace:
I20260806 20:35:28.946806 16621 raft_consensus.cc:3032] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:28.947418 16621 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:28.939209 16514 raft_consensus.cc:3032] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:28.948879 16634 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.954137 16436 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:28.808992 16622 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:28.788717 16157 raft_consensus.cc:686] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:28.955279 14553 raft_consensus.cc:1226] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:35:28.888923 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.956003 14550 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:28.788875 16602 raft_consensus.cc:686] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:28.956111 16157 consensus_queue.cc:237] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.957491 16602 consensus_queue.cc:237] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:28.895936 14633 rpcz_store.cc:269] 0806 20:35:25.425254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.037167 (+1611913us) service_pool.cc:227] Handling call
0806 20:35:28.833946 (+1796779us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400012cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.834050 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2359950080}
I20260806 20:35:28.898586 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:28.960270 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 61735.7us
I20260806 20:35:28.960400 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.961036 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.901389 16631 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:28.961725 14626 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:28.961951 16631 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:28.962575 16631 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:28.901995 16399 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.890722 14474 rpcz_store.cc:276] 0806 20:35:27.695611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.230908 (+535297us) service_pool.cc:227] Handling call
0806 20:35:28.231295 (+ 387us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:28.277424 (+ 46129us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.277487 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:28.284592 (+ 7105us) log.cc:842] Serialized 52 byte log entry
0806 20:35:28.520951 (+236359us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:28.521009 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.521111 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:28.821405 (+300294us) raft_consensus.cc:1671] finished
0806 20:35:28.821425 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.821614 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9587,"raft.queue_time_us":24051,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":4271,"thread_start_us":169109,"threads_started":1,"wal-append.queue_time_us":282011}
I20260806 20:35:28.963801 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.905241 16545 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:28.964653 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:28.964743 16411 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:28.964959 16545 consensus_queue.cc:1046] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:28.965539 16411 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:28.965909 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.966207 16411 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:28.966614 16545 raft_consensus.cc:3032] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:28.967214 16545 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:28.905869 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:28.906200 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.892541 16460 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:28.916909 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1663) took 2096 ms. Trace:
I20260806 20:35:28.885993 14624 rpcz_store.cc:276] 0806 20:35:26.481015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.080844 (+1599829us) service_pool.cc:227] Handling call
0806 20:35:28.252444 (+171600us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.353100 (+100656us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.823788 (+470688us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54002c1350. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.824010 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":240588544}
I20260806 20:35:28.917809 16486 raft_consensus.cc:3032] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:28.918834 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:28.919273 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:28.921087 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 236) took 1549 ms. Trace:
I20260806 20:35:28.901991 16309 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:28.803809 14442 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:28.946843 14478 rpcz_store.cc:269] 0806 20:35:27.427001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.496407 (+ 69406us) service_pool.cc:227] Handling call
0806 20:35:28.946298 (+1449891us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b5400328054. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.946401 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1228928768}
I20260806 20:35:28.950876 14518 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:28.902288 14471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1580) took 3949 ms (client timeout 2654 ms). Trace:
I20260806 20:35:28.951174 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:28.942349 16280 consensus_queue.cc:260] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.952203 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.952765 16514 raft_consensus.cc:729] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:28.962705 16285 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:28.950824 15946 consensus_meta.cc:287] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.514s user 0.007s sys 0.003s
I20260806 20:35:28.970739 14630 rpcz_store.cc:276] 0806 20:35:26.820883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.217718 (+1396835us) service_pool.cc:227] Handling call
0806 20:35:28.235294 (+ 17576us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:28.235784 (+ 490us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.235828 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:28.264126 (+ 28298us) log.cc:842] Serialized 26 byte log entry
0806 20:35:28.433821 (+169695us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:28.433861 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.433951 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:28.916704 (+482753us) raft_consensus.cc:1671] finished
0806 20:35:28.916723 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.916849 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":612,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":14860,"thread_start_us":143041,"threads_started":1,"wal-append.queue_time_us":333620}
I20260806 20:35:28.866734 14627 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:28.951963 16262 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.930900 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.981266 14444 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:29.061654 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 80474.1us
I20260806 20:35:29.042009 14555 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:29.061849 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.062829 14444 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.062716 14630 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:35:28.978878 14471 rpcz_store.cc:269] 0806 20:35:24.956324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:24.984002 (+ 27678us) service_pool.cc:227] Handling call
0806 20:35:28.902036 (+3918034us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54002170d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.902155 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3967305600}
I20260806 20:35:29.060088 14470 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:28.977900 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:28.977795 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1590) took 2117 ms. Trace:
W20260806 20:35:28.976111 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 172407us
I20260806 20:35:28.937811 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:28.977463 14518 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.034224 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.066721 14478 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:29.066838 14474 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:35:28.974227 14475 rpcz_store.cc:276] 0806 20:35:27.371241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.222312 (+851071us) service_pool.cc:227] Handling call
0806 20:35:28.307696 (+ 85384us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.354202 (+ 46506us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.920835 (+566633us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b54003c4950. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:28.921014 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195517824}
I20260806 20:35:29.065300 14476 rpcz_store.cc:276] 0806 20:35:26.860442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:26.860686 (+ 244us) service_pool.cc:227] Handling call
0806 20:35:26.872590 (+ 11904us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:27.259110 (+386520us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.977584 (+1718474us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54002166d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:28.977742 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":877455104}
I20260806 20:35:29.068071 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.068557 14475 raft_consensus.cc:1226] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:29.069202 14478 raft_consensus.cc:2379] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:28.972756 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1342) took 1549 ms. Trace:
I20260806 20:35:29.023214 16631 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.008718 16621 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.033682 16309 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:29.025004 16525 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:29.020200 16122 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.039143 16622 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.029685 16494 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:28.968417 16545 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:28.972900 16486 raft_consensus.cc:729] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:29.041533 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.103400 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 251) took 1643 ms. Trace:
W20260806 20:35:29.069416 14547 consensus_meta.cc:287] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.509s user 0.011s sys 0.000s
I20260806 20:35:29.103757 14595 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:28.973471 14629 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:28.884825 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1340) took 1508 ms. Trace:
I20260806 20:35:29.104769 14444 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:29.094774 14478 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:35:28.974100 14546 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:29.066948 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 89559.8us
I20260806 20:35:28.961726 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.106787 14478 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:29.079667 15151 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.107570 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:28.970597 16460 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.108546 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:29.108830 16488 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:29.109148 16460 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:29.109215 16488 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:29.109747 16488 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:29.109870 14469 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:29.111120 14469 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:28.974210 15180 raft_consensus.cc:1226] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:29.112516 16122 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.112811 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 488) took 1973 ms. Trace:
I20260806 20:35:29.113535 16545 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.113754 16494 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.103787 14630 rpcz_store.cc:276] 0806 20:35:27.460291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.063764 (+1603473us) service_pool.cc:227] Handling call
0806 20:35:29.102913 (+ 39149us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.103079 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:29.103101 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:29.103136 (+ 35us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:29.103164 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.103243 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:29.103348 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.114533 16486 consensus_queue.cc:260] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.115000 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.114761 16514 consensus_queue.cc:260] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.103029 16566 consensus_queue.cc:1046] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260806 20:35:29.074150 16631 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:29.086164 16436 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:29.070194 15182 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 356) took 1923 ms (client timeout 1894 ms). Trace:
I20260806 20:35:29.116508 16436 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:29.116778 15182 rpcz_store.cc:269] 0806 20:35:27.146521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.167240 (+ 20719us) service_pool.cc:227] Handling call
0806 20:35:29.069843 (+1902603us) spinlock_profiling.cc:243] Waited 88.9 ms on lock 0x7b540044e154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:29.070069 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204553600}
I20260806 20:35:29.117274 16623 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:29.077610 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:29.117873 16623 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.118479 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 498) took 1296 ms. Trace:
I20260806 20:35:29.118585 16436 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.118862 15185 rpcz_store.cc:276] 0806 20:35:27.821723 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.849416 (+ 27693us) service_pool.cc:227] Handling call
0806 20:35:29.077353 (+1227937us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:29.118421 (+ 41068us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43358080}
I20260806 20:35:29.076093 16532 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:29.113251 15180 rpcz_store.cc:276] 0806 20:35:27.138970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.144083 (+ 5113us) service_pool.cc:227] Handling call
0806 20:35:28.973858 (+1829775us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593bce 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:28.974012 (+ 154us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.112577 (+138565us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.112756 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32177024}
I20260806 20:35:29.104652 16465 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:29.122107 15151 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.122445 16309 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:29.123466 16628 raft_consensus.cc:3032] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:29.123916 16628 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:29.088042 16262 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:29.126396 16593 consensus_meta.cc:287] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.634s user 0.009s sys 0.000s
I20260806 20:35:29.126829 16593 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.127987 14632 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:29.128772 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.129673 16622 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.075098 16525 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:29.130160 14476 raft_consensus.cc:1226] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:29.130213 16622 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:29.070842 14624 rpcz_store.cc:276] 0806 20:35:27.423377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.971472 (+1548095us) service_pool.cc:227] Handling call
0806 20:35:28.971952 (+ 480us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.972236 (+ 284us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.972297 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.972373 (+ 76us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:28.972414 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.972535 (+ 121us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.972673 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.130906 16622 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.131538 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.132175 14476 raft_consensus.cc:3032] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:29.144870 14476 raft_consensus.cc:729] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:29.132705 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.160498 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.161530 14547 raft_consensus.cc:2379] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:29.134668 14441 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:29.193024 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 58432.1us
I20260806 20:35:29.135214 16494 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:29.122457 16411 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.136047 14552 raft_consensus.cc:1261] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:29.136647 16566 consensus_queue.cc:1225] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Peer 8e0cfd35f2994f91aaae46c25c6d6bff log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:35:29.136694 16384 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:29.136986 14518 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:29.129311 16415 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.115765 16486 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.203070 16415 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.138888 14470 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:29.129336 14471 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.142937 14595 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:29.208390 16472 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:29.209146 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 66286.7us
I20260806 20:35:29.144678 14442 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.071345 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:29.093331 14626 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:29.212440 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1338) took 1843 ms (client timeout 2222 ms). Trace:
W20260806 20:35:29.212617 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 68051.2us
W20260806 20:35:29.212863 14626 rpcz_store.cc:269] 0806 20:35:27.369284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.012822 (+643538us) service_pool.cc:227] Handling call
0806 20:35:28.960867 (+948045us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:29.212378 (+251511us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3668352}
I20260806 20:35:29.213876 14442 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.214032 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.144155 14521 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.137456 16630 consensus_queue.cc:1046] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:29.221257 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.134594 14632 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:29.162660 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:29.105201 14631 rpcz_store.cc:276] 0806 20:35:27.376445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.618797 (+1242352us) service_pool.cc:227] Handling call
0806 20:35:28.645052 (+ 26255us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.645319 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:28.744994 (+ 99675us) log.cc:842] Serialized 14 byte log entry
0806 20:35:28.884272 (+139278us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.884325 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:28.884371 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.884431 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:28.884753 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27013,"prepare.queue_time_us":136977,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":2585,"raft.queue_time_us":84,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":7861,"thread_start_us":218199,"threads_started":2,"wal-append.queue_time_us":194556}
I20260806 20:35:29.086339 16572 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.165414 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.173448 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.223489 14633 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:29.223809 16566 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:29.224578 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.224731 16554 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:29.225293 14633 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:29.226364 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.226881 14633 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.175522 14630 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:29.142058 16525 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.139201 16623 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.176249 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:29.240654 14551 raft_consensus.cc:2354] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:29.179270 14596 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.163116 16456 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:29.158916 16122 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.179813 14629 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:29.161036 16436 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:29.190259 14553 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:29.251917 15153 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:29.164415 14624 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:29.177232 16624 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:29.214887 16357 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:29.159665 14478 raft_consensus.cc:1261] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:29.256170 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.204254 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.258255 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.258806 14470 raft_consensus.cc:2379] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:29.259470 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.260022 14470 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:29.221813 16547 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.219309 14471 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:29.145895 14476 consensus_queue.cc:260] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.261938 14476 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.262041 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.227458 14479 raft_consensus.cc:1261] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:29.209012 16472 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.229535 16333 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.136375 16309 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:29.222102 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:29.194859 16622 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.239050 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.268093 14626 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 1.
I20260806 20:35:29.248322 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.250811 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:29.251001 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 106981us
I20260806 20:35:29.187604 16631 consensus_queue.cc:1046] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
W20260806 20:35:29.241341 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 104424us
I20260806 20:35:29.088763 15946 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.269932 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:29.253427 16436 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:29.271770 16576 log.cc:924] Time spent T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 2.564s user 0.008s sys 0.009s
I20260806 20:35:29.271982 15946 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.185339 16628 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.211659 14595 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:29.254199 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 75017us
I20260806 20:35:29.251657 16465 raft_consensus.cc:2741] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:35:29.232829 15821 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:29.293529 14477 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:29.295511 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1599) took 2372 ms. Trace:
I20260806 20:35:29.323083 14596 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:29.281139 14553 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:29.284162 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.278932 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 274) took 2110 ms. Trace:
I20260806 20:35:29.325582 14554 rpcz_store.cc:276] 0806 20:35:27.168419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.168575 (+ 156us) service_pool.cc:227] Handling call
0806 20:35:27.168876 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.400494 (+1231618us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.278684 (+878190us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54003e94d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:29.278873 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2830504192}
I20260806 20:35:29.277366 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.272872 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 242) took 2741 ms. Trace:
I20260806 20:35:29.329437 14628 rpcz_store.cc:276] 0806 20:35:26.531261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.105118 (+1573857us) service_pool.cc:227] Handling call
0806 20:35:28.470475 (+365357us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:28.470716 (+ 241us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.470755 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:28.470811 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:28.470860 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:28.470933 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:29.272694 (+801761us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b540009f450. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:29.272811 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":839448320}
I20260806 20:35:29.261045 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.269038 14631 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.269598 14521 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.270099 16440 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:29.251420 16566 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.273447 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:29.334319 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 64689.1us
W20260806 20:35:29.335578 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 66064.6us
I20260806 20:35:29.254727 16415 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:29.336889 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.343001 14521 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.338383 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.274076 16472 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.323675 14479 rpcz_store.cc:276] 0806 20:35:26.922945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:27.265825 (+342880us) service_pool.cc:227] Handling call
0806 20:35:27.266126 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.263070 (+1996944us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.295204 (+ 32134us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b54001077d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:29.295435 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4509932800}
I20260806 20:35:29.267398 16309 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:29.267331 16357 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.283245 16411 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.286870 16630 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:29.221207 16384 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.263145 16624 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.263415 16122 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.299523 16488 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.263690 16166 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:29.272367 16436 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.281421 16621 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.400913 14474 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:29.348413 14630 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
W20260806 20:35:29.351181 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 139591us
W20260806 20:35:29.286615 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 64598.5us
I20260806 20:35:29.403472 14595 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:29.274295 16514 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.356132 14479 raft_consensus.cc:1261] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:35:29.363085 14477 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:35:29.394887 14476 raft_consensus.cc:1261] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 20:35:29.350360 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.343636 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.384503 14518 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.386562 14470 raft_consensus.cc:1226] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:35:29.349187 14549 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:29.283620 16593 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.333010 16456 raft_consensus.cc:686] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:29.277064 16631 raft_consensus.cc:3032] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:29.355255 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.353960 14478 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:29.342173 16486 consensus_queue.cc:1046] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.134s
I20260806 20:35:29.476759 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.371405 16384 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:29.477180 16486 raft_consensus.cc:3032] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:29.356052 14521 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.242455 14442 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:29.477921 16486 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:29.431250 14595 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.479221 16473 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:29.479337 16593 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.479840 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:29.480532 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 49407.7us
I20260806 20:35:29.481462 14595 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.482401 16384 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:29.482573 15183 raft_consensus.cc:1226] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:29.339450 14633 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:29.484828 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:29.338918 16524 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:29.415084 14596 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.356659 16623 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:29.477676 14478 raft_consensus.cc:2354] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:29.383786 16628 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.479965 16473 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:29.511617 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:29.511823 14372 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45954 (request call id 234) took 14823 ms (client timeout 15000 ms). Trace:
I20260806 20:35:29.412707 14632 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:29.512349 14478 raft_consensus.cc:2354] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:29.511756 16473 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.513224 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.514370 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.515518 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.516287 14478 raft_consensus.cc:2379] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:29.364948 16542 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:29.472419 16411 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:29.527577 16542 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:29.487941 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:29.528928 15186 raft_consensus.cc:2354] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:29.496765 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:29.528262 16542 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.523721 16473 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.375113 16622 consensus_queue.cc:1046] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.158s
I20260806 20:35:29.533641 16622 raft_consensus.cc:3032] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:29.534198 16622 raft_consensus.cc:729] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:29.534541 16473 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.535411 16622 consensus_queue.cc:260] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.499353 14628 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:29.500039 14626 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.499796 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.556806 14633 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:29.361272 16572 raft_consensus.cc:3032] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:29.474953 14476 pending_rounds.cc:76] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 1
W20260806 20:35:29.495558 16641 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.138s user 0.000s sys 0.002s
I20260806 20:35:29.360920 16408 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:29.353952 16415 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:29.478509 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 122521us
I20260806 20:35:29.569216 14473 raft_consensus.cc:1226] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:29.569377 14521 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.373699 16280 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:29.490471 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:29.574234 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1614) took 1544 ms. Trace:
I20260806 20:35:29.574980 16415 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.507202 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.574872 14368 catalog_manager.cc:5495] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 2 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:29.577733 14477 raft_consensus.cc:1261] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:29.508776 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.578208 14368 catalog_manager.cc:5495] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 2 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:29.579620 14479 raft_consensus.cc:2354] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:29.401191 14627 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:29.580669 16524 raft_consensus.cc:2741] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:35:29.581182 14627 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:29.581875 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 254) took 1680 ms. Trace:
I20260806 20:35:29.582482 14521 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:29.521392 16532 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:29.521304 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 106350us
I20260806 20:35:29.363817 16357 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.522066 16593 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:29.522379 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:29.488281 16631 raft_consensus.cc:729] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:29.360630 16309 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:29.524866 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 282468us
I20260806 20:35:29.498138 16285 consensus_queue.cc:1046] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.109s
I20260806 20:35:29.479374 16486 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.498800 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:29.525540 16303 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:29.494150 16384 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:29.512372 14372 rpcz_store.cc:269] 0806 20:35:14.688727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:14.689895 (+ 1168us) service_pool.cc:227] Handling call
0806 20:35:15.174118 (+484223us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b5400002dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b944dd6c 00007fb9b9547c3e 00007fb9b7859fb4 00007fb9b7859f42 00007fb9b7859e71 00007fb9b7859dec 00007fb9b78590a2 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:29.511724 (+14337606us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:26.479541 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:26.605497 (+125956us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:26.618747 (+ 13250us) write_op.cc:435] Acquired schema lock
0806 20:35:26.638755 (+ 20008us) tablet.cc:660] Decoding operations
0806 20:35:26.838895 (+200140us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:26.839026 (+ 131us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:26.839056 (+ 30us) tablet.cc:683] Acquiring locks for 14 operations
0806 20:35:26.840380 (+ 1324us) tablet.cc:699] Row locks acquired
0806 20:35:26.840424 (+ 44us) write_op.cc:260] PREPARE: finished
0806 20:35:26.840654 (+ 230us) write_op.cc:270] Start()
0806 20:35:26.840816 (+ 162us) write_op.cc:276] Timestamp: P: 1786048526840578 usec, L: 0
0806 20:35:26.840852 (+ 36us) op_driver.cc:331] REPLICATION: starting
0806 20:35:26.919382 (+ 78530us) log.cc:842] Serialized 5667 byte log entry
0806 20:35:27.840966 (+921584us) op_driver.cc:444] REPLICATION: finished
0806 20:35:27.843502 (+ 2536us) write_op.cc:301] APPLY: starting
0806 20:35:27.843636 (+ 134us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:27.844690 (+ 1054us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:27.844868 (+ 178us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:27.847353 (+ 2485us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:27.872799 (+ 25446us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0806 20:35:27.882685 (+ 9886us) write_op.cc:312] APPLY: finished
0806 20:35:27.897222 (+ 14537us) log.cc:842] Serialized 101 byte log entry
0806 20:35:27.897506 (+ 284us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:27.897742 (+ 236us) write_op.cc:454] Released schema lock
0806 20:35:27.897843 (+ 101us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:27.898262 (+ 419us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:27.898305 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":995583,"reported_tablets":18,"spinlock_wait_cycles":939818752,"child_traces":[["op",{"apply.queue_time_us":2071,"apply.run_cpu_time_us":6512,"apply.run_wall_time_us":55480,"num_ops":14,"prepare.queue_time_us":901808,"prepare.run_cpu_time_us":65092,"prepare.run_wall_time_us":691748,"raft.queue_time_us":417507,"raft.run_cpu_time_us":1617,"raft.run_wall_time_us":6853,"replication_time_us":999974,"thread_start_us":834385,"threads_started":4,"wal-append.queue_time_us":417154,"wal-append.run_cpu_time_us":21699,"wal-append.run_wall_time_us":1571094}]]}
I20260806 20:35:29.526403 16525 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.538341 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:29.539494 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 252) took 2078 ms. Trace:
I20260806 20:35:29.606084 14632 rpcz_store.cc:276] 0806 20:35:27.461141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.343033 (+1881892us) service_pool.cc:227] Handling call
0806 20:35:29.412443 (+ 69410us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.539240 (+126797us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.539439 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.543107 16542 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.365509 16630 raft_consensus.cc:686] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:29.617015 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54494 (request call id 818) took 2193 ms. Trace:
I20260806 20:35:29.400559 16440 raft_consensus.cc:686] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:29.547643 14624 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:29.476831 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:29.558101 15153 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:29.625252 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 67255.6us
I20260806 20:35:29.625471 14555 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:29.600104 16415 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.559598 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1681) took 1749 ms. Trace:
I20260806 20:35:29.562059 14444 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:29.626613 14633 rpcz_store.cc:276] 0806 20:35:27.810449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.558552 (+1748103us) service_pool.cc:227] Handling call
0806 20:35:29.559044 (+ 492us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.559243 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:29.559277 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:29.559326 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:29.559364 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.559424 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:29.559539 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26112}
I20260806 20:35:29.501611 14595 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:29.628322 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 126785us
I20260806 20:35:29.628849 14553 raft_consensus.cc:595] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.629206 14553 raft_consensus.cc:603] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:29.629117 14595 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.381866 16488 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:29.425325 14630 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:35:29.632907 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1678) took 2249 ms. Trace:
I20260806 20:35:29.560597 14476 pending_rounds.cc:97] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:35:29.633505 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1344) took 2125 ms. Trace:
I20260806 20:35:29.633754 14476 raft_consensus.cc:2866] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:29.633870 14630 rpcz_store.cc:276] 0806 20:35:27.508238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.402419 (+1894181us) service_pool.cc:227] Handling call
0806 20:35:29.405842 (+ 3423us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:29.633202 (+227360us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.633442 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.560120 16572 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:29.469620 16436 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:29.489900 16639 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.118s user 0.000s sys 0.002s
W20260806 20:35:29.473102 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 88684.9us
I20260806 20:35:29.497030 16494 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:29.645265 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 83263.4us
I20260806 20:35:29.647137 14444 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:29.649945 14518 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.625182 16303 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260806 20:35:29.565135 14626 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:29.652773 14626 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260806 20:35:29.653192 14626 pending_rounds.cc:76] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 4
I20260806 20:35:29.653443 14626 pending_rounds.cc:97] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 20:35:29.653714 14626 raft_consensus.cc:2866] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:29.654109 16633 raft_consensus.cc:2741] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:35:29.654947 16633 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:29.655515 16633 raft_consensus.cc:686] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:29.574641 14473 rpcz_store.cc:276] 0806 20:35:28.029432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.467074 (+437642us) service_pool.cc:227] Handling call
0806 20:35:28.467494 (+ 420us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.569676 (+1102182us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.573958 (+ 4282us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003327d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:29.574158 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2533314432}
I20260806 20:35:29.582211 14627 rpcz_store.cc:276] 0806 20:35:27.901187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.580690 (+1679503us) service_pool.cc:227] Handling call
0806 20:35:29.581020 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.581656 (+ 636us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.581824 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.587819 14596 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.584430 14368 catalog_manager.cc:5495] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 3 to 4, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 4 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:29.415453 16166 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:29.665539 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 77754us
I20260806 20:35:29.595681 14521 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.602141 14628 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [3.2-5.4]
I20260806 20:35:29.414209 16465 consensus_queue.cc:1046] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.187s
I20260806 20:35:29.588354 16622 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.667243 14596 leader_election.cc:398] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Vote denied by peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) with higher term. Message: Invalid argument: T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:29.667542 14596 leader_election.cc:401] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 20:35:29.669266 14471 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:29.669375 16593 raft_consensus.cc:3032] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:29.669840 16593 raft_consensus.cc:729] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:29.669871 16166 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.670620 16460 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:29.670706 16593 consensus_queue.cc:260] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.672011 16593 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.590973 16408 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:29.617412 14474 rpcz_store.cc:276] 0806 20:35:27.416271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.471188 (+2054917us) service_pool.cc:227] Handling call
0806 20:35:29.604996 (+133808us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.672647 16408 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:29.618772 14442 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.673216 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1346) took 2164 ms. Trace:
W20260806 20:35:29.673964 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 55296.9us
I20260806 20:35:29.674224 16635 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) with higher term. Message: Invalid argument: T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:29.674739 14442 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.674917 16635 consensus_queue.cc:1046] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:29.675648 15821 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:29.675907 14442 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.626168 14632 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:29.676716 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 256) took 1589 ms. Trace:
I20260806 20:35:29.676508 16460 consensus_queue.cc:1046] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:29.677057 14632 rpcz_store.cc:276] 0806 20:35:28.087030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.607063 (+1520033us) service_pool.cc:227] Handling call
0806 20:35:29.625804 (+ 18741us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.676491 (+ 50687us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.676664 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.677728 16460 raft_consensus.cc:3032] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:29.678117 16460 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:29.678395 14632 raft_consensus.cc:1261] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:35:29.679313 16542 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:29.679589 14632 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:29.678869 16460 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.631726 16630 consensus_queue.cc:237] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.561350 15946 consensus_queue.cc:1046] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260806 20:35:29.680576 16631 consensus_queue.cc:260] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.591574 16547 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.592805 16411 consensus_queue.cc:1035] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260806 20:35:29.681769 14629 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:29.589174 16473 consensus_queue.cc:1046] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 20:35:29.683547 14629 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:29.630290 14595 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.684000 16460 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.684890 16122 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:29.685047 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 54842.8us
I20260806 20:35:29.676278 15821 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:29.603979 16357 consensus_queue.cc:1046] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260806 20:35:29.641965 14633 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:29.685796 15821 raft_consensus.cc:686] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:29.686321 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1348) took 1248 ms. Trace:
I20260806 20:35:29.686676 14633 rpcz_store.cc:276] 0806 20:35:28.437817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.627723 (+1189906us) service_pool.cc:227] Handling call
0806 20:35:29.641595 (+ 13872us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.686082 (+ 44487us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.686266 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.686779 15821 consensus_queue.cc:237] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.688365 16640 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:29.688771 16640 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:29.689069 16640 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.673667 14628 rpcz_store.cc:276] 0806 20:35:27.508945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.558597 (+2049652us) service_pool.cc:227] Handling call
0806 20:35:29.558975 (+ 378us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:35:29.672015 (+113040us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.672979 (+ 964us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b5400050854. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:29.673159 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98649472}
I20260806 20:35:29.650204 16524 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:29.671245 16623 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:29.643739 16545 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.676051 16285 raft_consensus.cc:3032] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:29.737939 16285 raft_consensus.cc:729] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:29.739115 16631 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.738945 16285 consensus_queue.cc:260] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.681093 15946 raft_consensus.cc:3032] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:29.681813 16514 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:29.633324 14625 rpcz_store.cc:276] 0806 20:35:27.383688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:28.966930 (+1583242us) service_pool.cc:227] Handling call
0806 20:35:28.967286 (+ 356us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:28.967735 (+ 449us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:28.967778 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:28.968405 (+ 627us) log.cc:842] Serialized 26 byte log entry
0806 20:35:29.238289 (+269884us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:29.238344 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.238428 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:29.494864 (+256436us) spinlock_profiling.cc:243] Waited 56 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:29.632611 (+137747us) raft_consensus.cc:1671] finished
0806 20:35:29.632634 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:29.632849 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1042,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":128813952,"thread_start_us":198281,"threads_started":1,"wal-append.queue_time_us":317440}
I20260806 20:35:29.740235 16285 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.375018 16624 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:29.749846 15185 raft_consensus.cc:1226] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:29.692350 14629 raft_consensus.cc:1261] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:35:29.753094 14625 raft_consensus.cc:1226] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:29.753410 14633 raft_consensus.cc:1226] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:29.636474 14630 raft_consensus.cc:1226] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:29.673134 16408 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:29.696878 14627 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:29.703517 16486 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.703612 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:29.709364 14442 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.709478 16303 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260806 20:35:29.758895 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 49604.6us
I20260806 20:35:29.760170 14442 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:29.635440 16398 consensus_queue.cc:1046] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.077s
I20260806 20:35:29.712514 16633 consensus_queue.cc:237] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.719249 16415 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.729261 14469 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:29.729390 16587 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:29.759495 16166 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:29.663730 16436 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.730340 16357 consensus_queue.cc:1035] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:29.648298 16456 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.734426 16525 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.734444 16286 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:29.735219 16572 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.664083 16384 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:29.622303 16628 consensus_queue.cc:1046] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260806 20:35:29.685801 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.751610 14553 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:29.752611 16566 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.693113 14632 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:29.754519 14625 raft_consensus.cc:1261] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:29.634009 16396 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:29.814982 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:29.816160 14549 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:29.820187 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.821275 14550 raft_consensus.cc:2379] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:29.755316 14633 raft_consensus.cc:1261] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:29.727990 14631 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:29.665908 16488 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:29.756773 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1349) took 1318 ms. Trace:
I20260806 20:35:29.709316 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 257) took 1066 ms. Trace:
I20260806 20:35:29.762044 14629 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:35:29.749993 15946 raft_consensus.cc:729] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:29.641374 16440 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.763499 16602 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:29.809552 14553 raft_consensus.cc:595] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.846395 14553 raft_consensus.cc:603] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:29.785446 16415 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:29.807941 16624 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:29.796972 16472 consensus_queue.cc:1046] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260806 20:35:29.811793 16473 raft_consensus.cc:3032] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
W20260806 20:35:29.826428 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 137161us
I20260806 20:35:29.826167 14630 rpcz_store.cc:276] 0806 20:35:28.438165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.636024 (+1197859us) service_pool.cc:227] Handling call
0806 20:35:29.636308 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.754087 (+117779us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.756700 (+ 2613us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:29.835175 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:29.835791 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 73767.2us
I20260806 20:35:29.789731 16587 raft_consensus.cc:686] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:29.844167 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.824643 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.841833 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.862118 14518 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:29.797374 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 111646us
W20260806 20:35:29.847246 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 85460.4us
I20260806 20:35:29.813038 16396 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:29.863626 14595 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.866258 14595 leader_election.cc:398] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:29.866639 14595 leader_election.cc:401] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260806 20:35:29.867084 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:29.860272 14549 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:29.872586 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.874704 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.876951 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:29.880745 14596 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.887450 14595 leader_election.cc:398] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:29.887867 14595 leader_election.cc:401] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:29.891328 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.892926 14595 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.796767 16286 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.815492 16514 consensus_queue.cc:1046] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260806 20:35:29.903596 14596 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.903280 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:29.903352 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:29.909108 14549 raft_consensus.cc:2362] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5. Re-sending same reply.
I20260806 20:35:29.909435 14547 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:29.894233 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:29.894405 14518 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:29.913579 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:29.914592 14552 raft_consensus.cc:2379] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:29.938172 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 44003.3us
I20260806 20:35:29.858522 16473 raft_consensus.cc:729] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:29.868317 16396 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:29.960763 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:29.961207 16645 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260806 20:35:29.891991 16593 raft_consensus.cc:2741] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:29.969981 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:29.847102 16488 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:29.961598 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:30.075796 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 219705us
I20260806 20:35:30.130630 16166 raft_consensus.cc:686] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:30.176230 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 307634us
W20260806 20:35:30.196214 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 424439us
I20260806 20:35:30.001298 16654 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.200s
I20260806 20:35:30.044283 16587 consensus_queue.cc:237] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:29.912107 16262 consensus_queue.cc:1046] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:29.941880 15946 consensus_queue.cc:260] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.216703 16122 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:30.217362 16623 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:29.898030 16624 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:30.218798 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:30.220315 14471 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:29.908020 16635 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:35:30.221576 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 625899us
I20260806 20:35:30.294766 14521 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:30.222452 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 318932us
I20260806 20:35:30.223306 16602 raft_consensus.cc:686] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:30.221628 14442 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:30.273334 16286 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:29.903471 16622 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:29.868335 16524 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:29.841815 14624 rpcz_store.cc:276] 0806 20:35:28.642673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.664720 (+1022047us) service_pool.cc:227] Handling call
0806 20:35:29.708811 (+ 44091us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.709005 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:29.709031 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:29.709074 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:29.709102 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.709156 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:29.709263 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:30.247885 14629 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:30.225195 16411 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:35:30.242890 16460 consensus_meta.cc:287] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.551s user 0.013s sys 0.008s
I20260806 20:35:30.323282 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 256) took 1641 ms. Trace:
W20260806 20:35:30.244704 16630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.003s sys 0.012s
W20260806 20:35:30.324647 16630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.003s sys 0.013s
I20260806 20:35:30.324885 16488 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:30.235872 16472 raft_consensus.cc:3032] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:30.327261 16472 raft_consensus.cc:729] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:30.270450 15185 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:30.268178 16653 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:30.328576 16545 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:35:29.925087 16415 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:30.282193 15150 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:30.319118 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:30.319267 16631 consensus_meta.cc:287] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.580s user 0.014s sys 0.002s
I20260806 20:35:30.283912 16655 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 20:35:30.320254 16524 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.293672 16566 consensus_queue.cc:1046] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 20:35:30.319584 16436 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:30.330569 14628 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:30.323711 14475 rpcz_store.cc:276] 0806 20:35:28.682214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.101378 (+419164us) service_pool.cc:227] Handling call
0806 20:35:29.101644 (+ 266us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:29.251808 (+150164us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:29.251837 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:29.255531 (+ 3694us) log.cc:842] Serialized 26 byte log entry
0806 20:35:29.625163 (+369632us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:29.625203 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:29.634698 (+ 9495us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54001a63d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:29.634741 (+ 43us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:30.322914 (+688173us) raft_consensus.cc:1671] finished
0806 20:35:30.322983 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:30.323218 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3648,"raft.queue_time_us":3907,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5123,"spinlock_wait_cycles":312872704,"thread_start_us":355114,"threads_started":1,"wal-append.queue_time_us":511141}
W20260806 20:35:30.364584 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 82511.5us
I20260806 20:35:30.284497 15179 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:30.373136 16262 consensus_queue.cc:1035] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20260806 20:35:30.296844 16623 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.391317 16262 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:30.391994 15821 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:30.368083 14629 raft_consensus.cc:1226] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:30.392544 15821 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.284330 16593 consensus_queue.cc:1035] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260806 20:35:30.393905 16524 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:30.374625 16486 consensus_meta.cc:287] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.612s user 0.008s sys 0.002s
I20260806 20:35:30.394536 15821 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:30.394995 16486 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:30.395145 16524 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.324368 16460 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260806 20:35:30.382751 15153 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:30.384369 16668 consensus_queue.cc:1035] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260806 20:35:30.385179 14633 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: []
W20260806 20:35:30.385789 16555 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.150s user 0.016s sys 0.045s
I20260806 20:35:30.387271 16436 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:30.396479 14555 raft_consensus.cc:1261] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:30.395671 16486 raft_consensus.cc:686] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:30.415562 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 120827us
I20260806 20:35:30.373420 16488 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:30.391933 16623 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:30.398566 16525 raft_consensus.cc:3032] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:30.422093 14521 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
W20260806 20:35:30.413688 16555 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.213s user 0.016s sys 0.045s
I20260806 20:35:30.373383 16672 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:30.328181 16472 consensus_queue.cc:260] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.440099 14518 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:30.441846 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:30.410789 16673 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.459905 14521 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:30.453073 16456 consensus_queue.cc:1046] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260806 20:35:30.395750 16524 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:30.432646 16494 consensus_meta.cc:287] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.767s user 0.011s sys 0.000s
I20260806 20:35:30.296489 16473 consensus_queue.cc:260] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:30.435829 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 194222us
I20260806 20:35:30.452963 14552 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:30.474031 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:30.489455 14475 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:30.489673 14596 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:30.474304 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:30.475086 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:30.476468 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:30.439338 16525 raft_consensus.cc:729] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:30.451232 16488 raft_consensus.cc:686] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:30.487169 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:30.459625 16309 consensus_meta.cc:287] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.842s user 0.014s sys 0.019s
I20260806 20:35:30.492909 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:30.494199 14442 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:30.429664 16436 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.484202 16472 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:30.466441 16532 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:30.455427 16514 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.012s sys 0.021s
W20260806 20:35:30.433490 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 215947us
I20260806 20:35:30.519093 14477 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:30.490489 16623 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:30.484678 16653 raft_consensus.cc:686] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:30.458217 16655 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:30.539829 14475 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
W20260806 20:35:30.540741 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 80971.7us
I20260806 20:35:30.521188 16415 raft_consensus.cc:686] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:30.402189 16602 consensus_queue.cc:237] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.492246 15821 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.397907 16166 consensus_queue.cc:237] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:30.547518 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 53397.5us
I20260806 20:35:30.615193 14442 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:30.493150 16494 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:30.553632 14595 leader_election.cc:398] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
W20260806 20:35:30.533139 16285 consensus_meta.cc:287] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.766s user 0.013s sys 0.007s
I20260806 20:35:30.551584 16532 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:30.545812 16280 consensus_meta.cc:287] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.960s user 0.009s sys 0.010s
W20260806 20:35:30.559134 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 176477us
I20260806 20:35:30.567934 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:30.632237 14547 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:30.570694 14546 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:30.574051 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:30.633970 14633 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:30.575393 14629 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:30.575614 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:30.635464 14630 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:30.576661 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:30.577737 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:30.580124 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:30.580236 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:30.649039 14632 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:30.616917 14552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:30.649860 14552 raft_consensus.cc:595] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:30.560400 16640 consensus_meta.cc:287] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.871s user 0.014s sys 0.006s
W20260806 20:35:30.620312 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 67106.2us
I20260806 20:35:30.658787 14595 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:30.660681 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:30.556878 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:30.556337 16623 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:30.581656 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:30.675894 14626 raft_consensus.cc:2396] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:30.650295 14552 raft_consensus.cc:603] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:30.655736 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:30.458532 16672 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.646345 14628 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:30.543951 16309 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.618444 16566 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:30.657542 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:30.609427 16514 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:30.543870 16440 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20260806 20:35:30.671769 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:30.696271 14630 raft_consensus.cc:2396] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:30.665068 14471 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:30.672577 14595 leader_election.cc:398] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:30.699791 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:30.701561 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:30.670924 16285 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:30.740689 16285 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:30.741381 16635 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:30.681365 16622 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:30.742234 16646 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.308s user 0.000s sys 0.000s
I20260806 20:35:30.511719 16669 consensus_queue.cc:1046] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.144s
I20260806 20:35:30.743456 16669 consensus_queue.cc:1225] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Peer 92828dd6dbe74f29a04d4e702b502621 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:35:30.637578 16532 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:30.749372 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:30.478940 16673 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.666445 16554 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.665298 14624 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:30.655743 16494 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:30.556648 16486 consensus_queue.cc:237] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.754407 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:30.797329 14552 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:30.777891 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:30.783530 14470 raft_consensus.cc:2379] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:30.658632 16460 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.683166 16631 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:30.682857 16280 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.619110 15821 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.685616 16623 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:30.748487 16663 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.202s user 0.000s sys 0.000s
I20260806 20:35:30.703022 16572 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:30.681142 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:30.691186 16514 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:30.815091 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 199974us
W20260806 20:35:30.794070 16408 consensus_meta.cc:287] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.033s user 0.006s sys 0.006s
I20260806 20:35:30.742344 16622 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.771704 16635 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:30.839744 14547 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:30.850925 16666 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.852003 16686 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:30.785086 16668 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.859545 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:30.863554 14632 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:30.836094 14633 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:30.644263 16525 consensus_queue.cc:260] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.815963 16494 raft_consensus.cc:686] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:30.679206 16640 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.879602 14442 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:30.892529 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 490158us
I20260806 20:35:30.816478 16631 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:30.899895 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 266795us
I20260806 20:35:30.855572 16630 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.910710 14546 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:30.944147 14478 raft_consensus.cc:2379] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:30.908826 14547 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
W20260806 20:35:30.926946 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47463.7us
I20260806 20:35:30.908517 16680 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:30.777030 16488 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.933297 16456 raft_consensus.cc:3032] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:30.935019 16494 consensus_queue.cc:237] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.888036 16514 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:30.829918 16587 consensus_queue.cc:1035] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260806 20:35:30.909066 16524 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:30.942042 16555 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:30.839025 16280 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.802755 16415 consensus_queue.cc:237] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:30.918721 16624 consensus_meta.cc:287] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.616s user 0.005s sys 0.006s
I20260806 20:35:30.938519 16631 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:30.945459 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:30.816800 16460 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.943359 16640 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:30.956151 16436 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:30.946419 15185 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.618s user 0.010s sys 0.000s
I20260806 20:35:30.946259 14442 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:31.009986 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 63852.2us
I20260806 20:35:31.010797 14442 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:30.773425 16653 consensus_queue.cc:237] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:30.945540 16675 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:30.803243 16554 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.978446 16688 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260806 20:35:30.956190 16456 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:30.901907 16408 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.926842 16668 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.014052 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 268) took 1332 ms. Trace:
I20260806 20:35:30.945278 16658 consensus_queue.cc:1046] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260806 20:35:30.944365 16630 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.946381 16680 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.985453 16555 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:30.918313 16666 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.006156 16624 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:31.146067 14627 rpcz_store.cc:276] 0806 20:35:29.681218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.761487 (+ 80269us) service_pool.cc:227] Handling call
0806 20:35:29.776123 (+ 14636us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:29.776524 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:29.776554 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:29.776990 (+ 436us) log.cc:842] Serialized 26 byte log entry
0806 20:35:30.074709 (+297719us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:30.074770 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:30.074871 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:31.013775 (+938904us) raft_consensus.cc:1671] finished
0806 20:35:31.013821 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:31.013988 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":81,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8388,"thread_start_us":257161,"threads_started":1,"wal-append.queue_time_us":644478}
I20260806 20:35:31.074079 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:31.074735 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:31.159466 14629 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:31.007339 16672 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:31.192023 14477 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.025s sys 0.016s
I20260806 20:35:31.033090 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1631) took 2166 ms. Trace:
W20260806 20:35:31.035009 16682 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260806 20:35:31.201632 14477 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.560s user 0.025s sys 0.017s
W20260806 20:35:31.036082 16691 log.cc:924] Time spent T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 20:35:31.022349 16303 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:31.037395 16681 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:31.036788 16685 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260806 20:35:31.209643 14554 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:31.098874 16285 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.098292 16411 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.119664 16635 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.010391 15185 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:31.027832 16640 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:31.021682 16655 raft_consensus.cc:686] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:31.073506 16309 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:31.247558 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 93769.2us
I20260806 20:35:31.243466 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:31.197679 16436 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.201879 16624 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.266201 14631 raft_consensus.cc:1226] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:31.224965 16672 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.281996 14628 raft_consensus.cc:2379] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:31.283063 14518 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:31.329172 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 46015.1us
I20260806 20:35:31.330363 14518 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:31.257440 16303 raft_consensus.cc:686] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:31.264153 15821 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:31.337510 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:31.185343 16532 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.261670 16285 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.284955 16657 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:31.328047 16122 consensus_meta.cc:287] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.034s user 0.011s sys 0.003s
I20260806 20:35:31.285413 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 368) took 1362 ms. Trace:
I20260806 20:35:31.285604 16659 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.354086 14633 raft_consensus.cc:2396] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:31.378486 14546 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:31.279031 16622 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:31.369549 14521 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:31.326434 16652 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:31.325750 16672 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.267239 16411 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:31.401149 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 70861.6us
I20260806 20:35:31.434217 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:31.434900 16714 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.435138 14631 raft_consensus.cc:2379] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:31.435333 16532 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.435506 16714 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.438165 16714 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.438251 16554 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.417263 16657 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.440006 16696 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:31.440251 16555 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.441241 16657 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.439218 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:31.360783 16566 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:31.322152 16624 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.440656 16696 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.224716 14469 rpcz_store.cc:276] 0806 20:35:28.866745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.111904 (+245159us) service_pool.cc:227] Handling call
0806 20:35:29.112239 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:29.775406 (+663167us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:31.022248 (+1246842us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b54003331d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:31.033024 (+ 10776us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1418011392}
I20260806 20:35:31.285355 16660 consensus_queue.cc:1046] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.172s
I20260806 20:35:31.353983 16280 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:31.323618 16667 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
I20260806 20:35:31.351043 16683 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:31.468411 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 454889us
I20260806 20:35:31.242744 16456 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.322729 16436 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.497401 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:31.535789 14633 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
I20260806 20:35:31.389376 16122 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:31.409437 16717 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.412319 16460 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:31.258080 16681 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.272629 16635 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.442953 16640 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.372253 15185 rpcz_store.cc:276] 0806 20:35:29.916937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.924149 (+ 7212us) service_pool.cc:227] Handling call
0806 20:35:31.279266 (+1355117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:31.481890 16532 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:31.523936 16622 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.557539 16524 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:31.563632 16686 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.564607 16630 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.564832 16731 raft_consensus.cc:3032] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:31.618620 16456 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:31.446223 16652 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.422535 16659 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.354779 16655 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:31.650564 14552 consensus_meta.cc:287] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.853s user 0.010s sys 0.003s
I20260806 20:35:31.666288 14552 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:31.656868 14477 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:31.564225 16408 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.562835 16717 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:31.584915 16706 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:31.668195 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 291) took 1097 ms. Trace:
I20260806 20:35:31.658973 16659 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:31.608701 16640 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.670559 16677 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:31.671926 16734 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.673520 16728 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.677788 16681 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.522338 16303 consensus_queue.cc:237] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.681375 16635 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.686069 16436 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.687000 16524 raft_consensus.cc:686] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:31.882368 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:31.652832 16679 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.388s user 0.001s sys 0.001s
I20260806 20:35:31.703819 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1640) took 1229 ms. Trace:
I20260806 20:35:31.881464 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 290) took 1310 ms. Trace:
I20260806 20:35:32.104612 14547 rpcz_store.cc:276] 0806 20:35:30.570889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.657276 (+ 86387us) service_pool.cc:227] Handling call
0806 20:35:31.881222 (+1223946us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54000870d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:31.881383 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":350823808}
I20260806 20:35:32.104486 14473 rpcz_store.cc:276] 0806 20:35:30.474297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.474516 (+ 219us) service_pool.cc:227] Handling call
0806 20:35:31.073703 (+599187us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:31.176560 (+102857us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:31.176606 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:31.176662 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:31.176710 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:31.703514 (+526804us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54001a4fd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:31.703585 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:31.703745 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235861888}
I20260806 20:35:32.069777 15821 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:32.106298 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:32.101104 14521 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.020s sys 0.100s
W20260806 20:35:32.107426 14521 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.666s user 0.021s sys 0.100s
W20260806 20:35:32.108606 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 739114us
I20260806 20:35:32.110810 14521 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:31.799525 16514 consensus_meta.cc:287] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.821s user 0.006s sys 0.005s
I20260806 20:35:32.106503 15821 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.700897 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1432) took 1816 ms. Trace:
I20260806 20:35:32.088101 14552 rpcz_store.cc:276] 0806 20:35:30.571152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.678256 (+107104us) service_pool.cc:227] Handling call
0806 20:35:31.668116 (+989860us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.099597 14625 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:32.111155 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:32.111735 14627 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.067s sys 0.067s
I20260806 20:35:31.629787 16122 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:31.809784 14624 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.993s user 0.011s sys 0.003s
I20260806 20:35:31.549813 16683 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.175951 15821 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:31.696291 16396 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.427s user 0.008s sys 0.006s
I20260806 20:35:31.623899 16532 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:32.145099 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.005s sys 0.016s
W20260806 20:35:32.189369 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.006s sys 0.016s
W20260806 20:35:32.067595 16472 consensus_meta.cc:287] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.504s user 0.006s sys 0.005s
I20260806 20:35:32.127612 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1651) took 1115 ms. Trace:
I20260806 20:35:32.194332 14473 rpcz_store.cc:276] 0806 20:35:31.012659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.105837 (+1093178us) service_pool.cc:227] Handling call
0806 20:35:32.127536 (+ 21699us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:32.157825 14627 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s user 0.069s sys 0.072s
I20260806 20:35:32.113715 14555 rpcz_store.cc:276] 0806 20:35:29.884192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.908278 (+ 24086us) service_pool.cc:227] Handling call
0806 20:35:29.909242 (+ 964us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:30.417220 (+507978us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:31.700573 (+1283353us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b54000b2cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:31.700807 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1119540096}
I20260806 20:35:31.702942 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:32.200029 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:32.200795 14473 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:32.159016 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:32.061004 16673 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.874146 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1639) took 1400 ms. Trace:
I20260806 20:35:31.689473 16696 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.624487 16622 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:31.694269 16411 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:31.651152 16731 raft_consensus.cc:729] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:31.697535 16465 raft_consensus.cc:3032] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:32.184506 14521 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:32.250211 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 65830.8us
I20260806 20:35:32.158198 16652 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:31.695246 16285 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:32.265522 16285 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:32.266510 16285 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.099987 16635 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:32.061530 16280 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.179075 16677 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:32.069404 16357 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.801872 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1642) took 1325 ms. Trace:
W20260806 20:35:31.796497 16572 consensus_meta.cc:287] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.909s user 0.010s sys 0.004s
I20260806 20:35:32.180886 16122 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.174263 14624 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:32.199489 16384 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.137105 16514 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.206238 16593 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.126866 15946 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:32.208993 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:32.317289 14625 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:31.637509 16488 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.666697 16686 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.131150 16436 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:31.886636 16666 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:31.768350 16621 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:31.703296 16723 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:31.708292 16680 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:32.156520 16722 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:32.234875 15821 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:32.353428 15821 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:32.200438 16631 consensus_meta.cc:287] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.183s user 0.006s sys 0.007s
I20260806 20:35:32.242208 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:32.359400 14552 raft_consensus.cc:2379] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:32.359992 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1580) took 1357 ms. Trace:
I20260806 20:35:32.360579 14552 rpcz_store.cc:276] 0806 20:35:31.003054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.240000 (+1236946us) service_pool.cc:227] Handling call
0806 20:35:32.359905 (+119905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:31.886067 16715 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.251950 14521 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:32.373368 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 121514us
W20260806 20:35:32.092782 16486 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.019s sys 0.033s
I20260806 20:35:32.259694 16729 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.249306 16411 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:31.681396 16668 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:31.635701 16721 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.632s
I20260806 20:35:32.275432 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1650) took 1263 ms. Trace:
I20260806 20:35:32.209008 16672 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:32.277801 16532 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:32.280664 16676 log.cc:924] Time spent T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 1.302s user 0.001s sys 0.002s
I20260806 20:35:32.277880 16622 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:32.282059 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1368) took 1012 ms. Trace:
I20260806 20:35:32.282475 16683 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:32.283420 14633 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:35:32.333938 16488 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.253993 16524 consensus_queue.cc:237] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:32.286286 16623 consensus_meta.cc:287] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.408s user 0.006s sys 0.008s
I20260806 20:35:32.298426 14470 rpcz_store.cc:276] 0806 20:35:30.476009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.476169 (+ 160us) service_pool.cc:227] Handling call
0806 20:35:31.801614 (+1325445us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54002323d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:31.801780 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":645765504}
I20260806 20:35:32.205714 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:32.499420 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1578) took 2664 ms. Trace:
I20260806 20:35:32.242527 16681 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:32.306414 14624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1702) took 1729 ms (client timeout 2142 ms). Trace:
I20260806 20:35:32.298245 16677 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:32.306272 16384 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.301883 16357 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.307554 16593 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.309046 16122 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:32.318567 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1705) took 1739 ms (client timeout 2185 ms). Trace:
W20260806 20:35:32.524850 14625 rpcz_store.cc:269] 0806 20:35:30.579271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.882005 (+1302734us) service_pool.cc:227] Handling call
0806 20:35:32.318486 (+436481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.526279 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:32.527798 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:32.250321 16465 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:32.213954 14471 rpcz_store.cc:276] 0806 20:35:30.473925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.474078 (+ 153us) service_pool.cc:227] Handling call
0806 20:35:31.873812 (+1399734us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54001a54d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:31.874040 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":401005440}
I20260806 20:35:32.298523 16635 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:32.330799 16456 consensus_meta.cc:287] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.689s user 0.005s sys 0.006s
W20260806 20:35:32.338380 14520 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.006s sys 0.021s
I20260806 20:35:32.341971 16732 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:32.342382 14554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1577) took 2507 ms (client timeout 1853 ms). Trace:
I20260806 20:35:32.344825 14549 raft_consensus.cc:1226] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:32.211287 16659 consensus_queue.cc:1035] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260806 20:35:32.098430 16706 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.298354 16624 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:32.297565 16673 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:32.158625 16734 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.358093 16666 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:32.367908 16671 log.cc:924] Time spent T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.388s user 0.000s sys 0.003s
I20260806 20:35:32.368389 16631 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.361830 16309 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.323514 16628 raft_consensus.cc:3032] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:32.372403 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:32.598456 14552 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:35:32.379456 16698 log.cc:924] Time spent T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 1.375s user 0.000s sys 0.003s
I20260806 20:35:32.300323 16280 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.393285 16705 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:32.349601 16723 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:32.350234 16285 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:32.397046 15821 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:32.381062 16486 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.922s user 0.019s sys 0.034s
I20260806 20:35:32.192517 16728 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.381093 16436 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:32.430068 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1430) took 2550 ms. Trace:
I20260806 20:35:32.650202 14553 rpcz_store.cc:276] 0806 20:35:29.879644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.892557 (+ 12913us) service_pool.cc:227] Handling call
0806 20:35:29.892987 (+ 430us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:29.935019 (+ 42032us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:30.220443 (+285424us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:30.242729 (+ 22286us) log.cc:842] Serialized 52 byte log entry
0806 20:35:30.433810 (+191081us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:30.569142 (+135332us) log.cc:842] Serialized 14 byte log entry
0806 20:35:30.569484 (+ 342us) log.cc:842] Serialized 14 byte log entry
0806 20:35:30.569630 (+ 146us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:30.569701 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:32.429796 (+1860095us) raft_consensus.cc:1671] finished
0806 20:35:32.429823 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:32.429988 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":250547,"prepare.queue_time_us":1790816,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":100,"raft.queue_time_us":1325546,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":25846,"thread_start_us":246596,"threads_started":2,"wal-append.queue_time_us":854626}
I20260806 20:35:32.440933 14632 rpcz_store.cc:276] 0806 20:35:31.269912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.270069 (+ 157us) service_pool.cc:227] Handling call
0806 20:35:31.270405 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:31.522960 (+252555us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:31.523004 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:31.523058 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:31.523103 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:32.281705 (+758602us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b540039fdd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:32.281770 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:32.281973 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":170446,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"spinlock_wait_cycles":579866880}
I20260806 20:35:32.188761 16396 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.395546 16729 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.355130 16621 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.401721 16411 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.446190 16733 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:32.447739 16678 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:32.448468 16622 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.712368 15179 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:32.712783 16622 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.713973 16729 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:32.714983 16622 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:32.411500 14469 rpcz_store.cc:276] 0806 20:35:31.011760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.565643 (+553883us) service_pool.cc:227] Handling call
0806 20:35:32.275314 (+709671us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.715682 16621 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:32.492887 16713 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:32.494736 16697 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.496627 16473 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:32.500022 14551 rpcz_store.cc:276] 0806 20:35:29.835113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.835271 (+ 158us) service_pool.cc:227] Handling call
0806 20:35:29.855888 (+ 20617us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:29.856290 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:29.856330 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:29.902942 (+ 46612us) log.cc:842] Serialized 26 byte log entry
0806 20:35:30.421758 (+518816us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:30.421908 (+ 150us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:30.422197 (+ 289us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:32.279119 (+1856922us) spinlock_profiling.cc:243] Waited 8.79 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:32.440443 (+161324us) raft_consensus.cc:1671] finished
0806 20:35:32.440488 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:32.499322 (+ 58834us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":46840,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":38418,"spinlock_wait_cycles":20226432,"thread_start_us":181140,"threads_started":1,"wal-append.queue_time_us":763608}
I20260806 20:35:32.511072 14470 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:32.721001 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1498) took 2054 ms. Trace:
I20260806 20:35:32.721426 14470 rpcz_store.cc:276] 0806 20:35:30.666322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.510424 (+1844102us) service_pool.cc:227] Handling call
0806 20:35:32.510889 (+ 465us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:32.720706 (+209817us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:32.720936 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.510622 16623 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.511742 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 260) took 1469 ms. Trace:
I20260806 20:35:32.512281 16681 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.317845 16731 consensus_queue.cc:260] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.438251 16672 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.437898 16727 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:32.531052 14521 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:32.515776 14624 rpcz_store.cc:269] 0806 20:35:30.574652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.576294 (+ 1642us) service_pool.cc:227] Handling call
0806 20:35:32.303868 (+1727574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.532914 16647 raft_consensus.cc:3032] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:32.534106 16465 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.535390 16384 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:32.503288 16545 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:32.537143 14520 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.058s user 0.006s sys 0.022s
W20260806 20:35:32.537425 14554 rpcz_store.cc:269] 0806 20:35:29.834763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.834910 (+ 147us) service_pool.cc:227] Handling call
0806 20:35:32.342062 (+2507152us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54001fa9d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:32.342281 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3104631680}
W20260806 20:35:32.578224 14628 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.012s sys 0.069s
W20260806 20:35:32.891595 14628 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.262s user 0.012s sys 0.069s
I20260806 20:35:32.582522 14625 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:32.523376 16640 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:32.584034 15151 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.755s user 0.020s sys 0.027s
I20260806 20:35:32.864252 16286 raft_consensus.cc:3032] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:32.536290 16514 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:32.536648 16717 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:32.598784 14630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.013s sys 0.019s
W20260806 20:35:32.629045 14631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.036s user 0.011s sys 0.023s
W20260806 20:35:32.602653 14478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1643) took 2126 ms (client timeout 2080 ms). Trace:
W20260806 20:35:32.599094 14552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 293) took 2027 ms (client timeout 2690 ms). Trace:
I20260806 20:35:32.599781 14547 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:32.639755 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1715) took 1208 ms. Trace:
I20260806 20:35:32.382841 16722 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:32.641000 14477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1499) took 1973 ms (client timeout 2086 ms). Trace:
W20260806 20:35:32.941018 14477 rpcz_store.cc:269] 0806 20:35:30.667335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.438948 (+771613us) service_pool.cc:227] Handling call
0806 20:35:32.640740 (+1201792us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54002175d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:32.640906 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":493047552}
I20260806 20:35:32.925113 14633 rpcz_store.cc:276] 0806 20:35:31.431176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.536498 (+105322us) service_pool.cc:227] Handling call
0806 20:35:32.283086 (+746588us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:32.447066 (+163980us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:32.639445 (+192379us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b540039fdd0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:32.639672 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1659908864}
I20260806 20:35:32.591408 16673 raft_consensus.cc:686] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:32.655226 16357 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:32.655351 16734 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:32.655850 16396 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:32.634456 16624 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.705994 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:32.597890 16628 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:32.906068 16286 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:32.642778 16723 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:32.495517 16683 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.716799 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:32.719775 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.230s user 0.025s sys 0.033s
W20260806 20:35:32.976828 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.496s user 0.025s sys 0.033s
I20260806 20:35:32.648433 16705 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:32.598152 16666 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.731305 14473 rpcz_store.cc:276] 0806 20:35:31.042143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.201745 (+1159602us) service_pool.cc:227] Handling call
0806 20:35:32.511676 (+309931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.799680 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:32.981920 14551 raft_consensus.cc:2362] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7. Re-sending same reply.
W20260806 20:35:32.982642 14551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 299) took 1613 ms (client timeout 1937 ms). Trace:
W20260806 20:35:32.983073 14551 rpcz_store.cc:269] 0806 20:35:31.368930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.720297 (+1351367us) service_pool.cc:227] Handling call
0806 20:35:32.982548 (+262251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.546859 16732 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.780870 14469 raft_consensus.cc:1226] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:32.600912 16309 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:32.732371 14479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.353s user 0.009s sys 0.023s
W20260806 20:35:32.996781 14479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.617s user 0.009s sys 0.029s
W20260806 20:35:32.845726 15153 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.622s user 0.013s sys 0.018s
W20260806 20:35:32.997788 15153 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.774s user 0.014s sys 0.018s
I20260806 20:35:32.704188 16436 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:32.849119 16494 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.188s user 0.011s sys 0.032s
W20260806 20:35:32.865371 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 334422us
I20260806 20:35:32.891927 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:32.387295 16715 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.848915 16686 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:32.904102 15151 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.075s user 0.020s sys 0.027s
I20260806 20:35:32.843618 16678 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:32.925172 14478 rpcz_store.cc:269] 0806 20:35:30.476395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.476972 (+ 577us) service_pool.cc:227] Handling call
0806 20:35:32.602388 (+2125416us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54001a5ed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:32.602562 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986620672}
W20260806 20:35:32.928714 14631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.336s user 0.011s sys 0.023s
W20260806 20:35:32.928794 14630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.296s user 0.014s sys 0.019s
W20260806 20:35:32.928858 14552 rpcz_store.cc:269] 0806 20:35:30.571695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.371951 (+1800256us) service_pool.cc:227] Handling call
0806 20:35:32.599016 (+227065us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:32.871371 16740 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:33.025573 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:32.937891 14547 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:32.868186 16672 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.904551 16647 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:32.958751 15150 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.766s user 0.021s sys 0.038s
W20260806 20:35:33.029172 15150 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.837s user 0.021s sys 0.038s
W20260806 20:35:33.029570 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 837229us
I20260806 20:35:33.029665 15821 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:32.959282 14553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1442) took 1678 ms (client timeout 1996 ms). Trace:
I20260806 20:35:32.961580 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:33.030356 14553 rpcz_store.cc:269] 0806 20:35:31.280630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.705649 (+1425019us) service_pool.cc:227] Handling call
0806 20:35:32.959206 (+253557us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:33.031658 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:32.716859 16733 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.993119 16706 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:32.921923 16731 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:33.041702 14477 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:32.921566 16737 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.830061 16697 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.830404 16713 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:32.908864 16465 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:32.903981 16486 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.977017 14632 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:33.054009 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1376) took 1394 ms. Trace:
I20260806 20:35:33.054491 14632 rpcz_store.cc:276] 0806 20:35:31.660109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.652868 (+992759us) service_pool.cc:227] Handling call
0806 20:35:33.053913 (+401045us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:32.977201 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 1.4966e+06us
I20260806 20:35:33.036340 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 262) took 1993 ms. Trace:
W20260806 20:35:32.963956 15946 consensus_meta.cc:287] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.643s user 0.011s sys 0.003s
I20260806 20:35:32.950023 16488 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:33.078437 16488 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:33.079275 16488 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:33.079929 16525 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:32.953035 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:33.080821 14624 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:33.081751 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1375) took 1422 ms. Trace:
I20260806 20:35:33.081785 16657 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:33.082473 14624 rpcz_store.cc:276] 0806 20:35:31.659475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.905231 (+1245756us) service_pool.cc:227] Handling call
0806 20:35:33.081656 (+176425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:33.082746 16657 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.906628 16545 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:33.086803 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1378) took 1197 ms. Trace:
I20260806 20:35:33.087253 14625 rpcz_store.cc:276] 0806 20:35:31.889819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.527539 (+637720us) service_pool.cc:227] Handling call
0806 20:35:33.086612 (+559073us) spinlock_profiling.cc:243] Waited 53.7 ms on lock 0x7b540006efd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:33.086733 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123611520}
I20260806 20:35:33.060817 16486 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:33.090464 16486 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:33.091147 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:33.051374 16741 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.333s user 0.000s sys 0.002s
I20260806 20:35:32.962411 16593 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:32.997370 15187 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.748s user 0.020s sys 0.024s
W20260806 20:35:32.968787 16587 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.021s sys 0.040s
I20260806 20:35:33.013306 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:32.885975 16727 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:32.987792 16742 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.989784 16705 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:32.986368 16280 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:33.030776 14633 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:32.710143 14626 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.094s user 0.017s sys 0.015s
I20260806 20:35:32.987639 16631 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:33.002171 16628 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:33.035615 14553 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:32.956702 16633 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.634s user 0.019s sys 0.043s
W20260806 20:35:33.008337 16494 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.348s user 0.012s sys 0.032s
I20260806 20:35:33.003821 16396 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:32.966540 16624 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:33.026993 16673 consensus_queue.cc:237] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:33.084412 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:33.190999 14624 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:33.191174 16728 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:33.191586 16733 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:33.036696 15821 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:33.021781 16411 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:32.988008 16683 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:33.038342 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.628s user 0.013s sys 0.060s
W20260806 20:35:32.964370 16739 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
W20260806 20:35:33.042698 14444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.752s user 0.007s sys 0.028s
W20260806 20:35:33.066625 14518 connection.cc:232] RPC call timeout handler was delayed by -1.525242134986911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
I20260806 20:35:33.069612 16666 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:33.038784 16286 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:33.060523 14469 rpcz_store.cc:276] 0806 20:35:31.042795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.715821 (+1673026us) service_pool.cc:227] Handling call
0806 20:35:32.716188 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:33.005091 (+288903us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:33.032426 (+ 27335us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:33.088310 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:33.244267 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.025s sys 0.072s
I20260806 20:35:33.046125 16623 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:33.045444 16740 raft_consensus.cc:686] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:33.190316 16748 log.cc:924] Time spent T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260806 20:35:33.199792 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 284) took 2632 ms. Trace:
I20260806 20:35:33.323256 14549 rpcz_store.cc:276] 0806 20:35:30.567753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.567906 (+ 153us) service_pool.cc:227] Handling call
0806 20:35:32.344585 (+1776679us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:32.538780 (+194195us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:33.199493 (+660713us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54001fa9d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:33.199701 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4085312896}
I20260806 20:35:33.099821 14474 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:33.097076 14552 raft_consensus.cc:2379] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:33.192094 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:33.274487 15187 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.031s user 0.027s sys 0.035s
I20260806 20:35:33.326013 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 521) took 2677 ms. Trace:
I20260806 20:35:32.958348 16635 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:33.275550 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.027s sys 0.073s
I20260806 20:35:33.331054 14554 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:33.276960 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.867s user 0.018s sys 0.064s
I20260806 20:35:33.230212 15821 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:33.224713 14546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 295) took 2652 ms (client timeout 2177 ms). Trace:
W20260806 20:35:33.278308 14444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.988s user 0.007s sys 0.028s
I20260806 20:35:33.336756 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:33.279086 14553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1449) took 1708 ms (client timeout 2115 ms). Trace:
I20260806 20:35:33.104154 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:33.283547 14518 connection.cc:232] RPC call timeout handler was delayed by -1.2687421349983197s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
W20260806 20:35:33.286667 14551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1443) took 2005 ms (client timeout 2206 ms). Trace:
W20260806 20:35:33.290391 14471 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.007s sys 0.031s
W20260806 20:35:33.053975 16415 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.349s user 0.027s sys 0.058s
I20260806 20:35:33.055928 16681 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:33.117934 16745 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260806 20:35:33.113200 16744 log.cc:924] Time spent T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260806 20:35:33.101621 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:33.279170 14626 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.663s user 0.023s sys 0.018s
W20260806 20:35:33.279870 16587 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.069s user 0.022s sys 0.046s
I20260806 20:35:33.050259 16737 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:33.332384 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:34.047202 14441 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.849s (SENT) [suppressed 11 similar messages]
I20260806 20:35:33.326668 15187 rpcz_store.cc:276] 0806 20:35:30.648133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.648327 (+ 194us) service_pool.cc:227] Handling call
0806 20:35:30.692860 (+ 44533us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:30.693139 (+ 279us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:31.134396 (+441257us) log.cc:842] Serialized 14 byte log entry
0806 20:35:33.325434 (+2191038us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:33.325506 (+ 72us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:33.325569 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:33.325668 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:33.325833 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38204,"prepare.queue_time_us":1407653,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":358476,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":32951,"thread_start_us":2508400,"threads_started":3,"wal-append.queue_time_us":516882,"wal-append.run_cpu_time_us":2137,"wal-append.run_wall_time_us":1006371}
W20260806 20:35:33.792835 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 258) took 3574 ms (client timeout 2220 ms). Trace:
W20260806 20:35:33.680935 14470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.011s sys 0.018s
W20260806 20:35:33.791957 14628 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.005s sys 0.017s
W20260806 20:35:34.049190 14470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.179s user 0.015s sys 0.044s
W20260806 20:35:34.049520 14628 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.120s user 0.008s sys 0.024s
I20260806 20:35:34.050288 14477 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
W20260806 20:35:33.365520 14443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.022s sys 0.023s
W20260806 20:35:33.365289 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.869s user 0.046s sys 0.048s
W20260806 20:35:33.337688 14553 rpcz_store.cc:269] 0806 20:35:31.571120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.031327 (+1460207us) service_pool.cc:227] Handling call
0806 20:35:33.279008 (+247681us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:33.961752 14479 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.928s user 0.004s sys 0.028s
W20260806 20:35:34.053633 14479 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.020s user 0.006s sys 0.029s
W20260806 20:35:33.337016 14546 rpcz_store.cc:269] 0806 20:35:30.572234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.945091 (+372857us) service_pool.cc:227] Handling call
0806 20:35:33.224424 (+2279333us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540013a6d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:33.224622 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":875350400}
W20260806 20:35:33.518810 14475 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s user 0.012s sys 0.031s
W20260806 20:35:33.891605 14478 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.016s sys 0.013s
I20260806 20:35:33.337740 14625 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
W20260806 20:35:34.052591 14551 rpcz_store.cc:269] 0806 20:35:31.280914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.992961 (+1712047us) service_pool.cc:227] Handling call
0806 20:35:33.286591 (+293630us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:34.055562 14478 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.018s user 0.022s sys 0.014s
W20260806 20:35:33.883697 14521 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.873s user 0.008s sys 0.016s
I20260806 20:35:34.055251 14632 raft_consensus.cc:2379] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:33.344758 15821 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:33.706509 14442 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.011s sys 0.016s
W20260806 20:35:34.057055 14442 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.104s user 0.019s sys 0.023s
I20260806 20:35:34.056926 15821 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:34.051502 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:33.871603 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.006s sys 0.031s
I20260806 20:35:33.637425 14554 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
W20260806 20:35:34.058115 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.020s user 0.015s sys 0.036s
W20260806 20:35:33.886865 15151 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.000s sys 0.023s
W20260806 20:35:34.058928 15151 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.039s user 0.003s sys 0.025s
W20260806 20:35:34.057477 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 1.10521e+06us
W20260806 20:35:34.056320 14521 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.046s user 0.013s sys 0.016s
W20260806 20:35:34.052254 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.643s user 0.052s sys 0.052s
W20260806 20:35:34.052626 14471 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.463s user 0.016s sys 0.050s
W20260806 20:35:34.060004 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 1.05001e+06us
W20260806 20:35:34.057241 14632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1720) took 1680 ms (client timeout 1822 ms). Trace:
W20260806 20:35:34.063475 14632 rpcz_store.cc:269] 0806 20:35:32.376553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.055311 (+678758us) service_pool.cc:227] Handling call
0806 20:35:34.057188 (+1001877us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:34.048938 14474 rpcz_store.cc:269] 0806 20:35:30.218400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.218543 (+ 143us) service_pool.cc:227] Handling call
0806 20:35:33.792533 (+3573990us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b54002af154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:33.792728 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6423815424}
I20260806 20:35:33.422556 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1367) took 2153 ms. Trace:
I20260806 20:35:34.066246 14627 rpcz_store.cc:276] 0806 20:35:31.269416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.273817 (+ 4401us) service_pool.cc:227] Handling call
0806 20:35:31.322621 (+ 48804us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:31.445909 (+123288us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:31.445954 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:31.459105 (+ 13151us) log.cc:842] Serialized 26 byte log entry
0806 20:35:32.195962 (+736857us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:32.196002 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:32.196069 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:33.422237 (+1226168us) raft_consensus.cc:1671] finished
0806 20:35:33.422269 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:33.422469 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":96255,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":34637,"thread_start_us":859245,"threads_started":2,"wal-append.queue_time_us":1188607}
W20260806 20:35:33.952348 15153 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.903s user 0.014s sys 0.006s
W20260806 20:35:34.068786 15153 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.057s user 0.014s sys 0.008s
W20260806 20:35:34.069321 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 1.05734e+06us
W20260806 20:35:34.051064 14443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.700s user 0.035s sys 0.040s
W20260806 20:35:34.055181 14475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.453s user 0.022s sys 0.044s
I20260806 20:35:34.071573 14633 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:34.072290 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1381) took 1826 ms. Trace:
I20260806 20:35:34.072796 14633 rpcz_store.cc:276] 0806 20:35:32.245786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.275902 (+1030116us) service_pool.cc:227] Handling call
0806 20:35:34.072218 (+796316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:34.065205 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:33.279959 16633 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.957s user 0.019s sys 0.050s
I20260806 20:35:33.302369 16486 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:34.079677 15151 connection.cc:232] RPC call timeout handler was delayed by -2.6591922110237647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
I20260806 20:35:33.483259 14624 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
W20260806 20:35:34.009433 16755 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.391s user 0.000s sys 0.000s
W20260806 20:35:33.817612 16730 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.004s sys 0.027s
W20260806 20:35:33.631227 16751 log.cc:924] Time spent T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.338s user 0.001s sys 0.001s
W20260806 20:35:34.039204 16754 log.cc:924] Time spent T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.345s user 0.000s sys 0.000s
W20260806 20:35:34.013231 16749 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
W20260806 20:35:33.354432 16653 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.657s user 0.017s sys 0.048s
I20260806 20:35:33.622429 16436 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:34.181905 16658 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:34.051522 16714 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:33.365957 16309 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:33.331893 16628 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:33.560037 16740 consensus_queue.cc:237] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:33.271747 16743 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:33.364823 16488 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:33.290746 16742 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:34.180435 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1726) took 1167 ms. Trace:
W20260806 20:35:34.179816 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 118623us
W20260806 20:35:34.185964 14442 connection.cc:232] RPC call timeout handler was delayed by -1.0387319659930654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
I20260806 20:35:34.187294 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:34.189188 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:34.187911 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:34.187345 14553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1454) took 1773 ms (client timeout 1621 ms). Trace:
I20260806 20:35:33.086861 16545 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:33.114807 16647 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:33.290506 16656 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:34.814002 14629 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.268s user 0.024s sys 0.053s
I20260806 20:35:34.889175 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1713) took 3458 ms. Trace:
W20260806 20:35:34.895260 15152 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.840s user 0.017s sys 0.032s
W20260806 20:35:35.031481 14555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1440) took 4011 ms (client timeout 1936 ms). Trace:
W20260806 20:35:34.815722 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1383) took 2569 ms (client timeout 2128 ms). Trace:
W20260806 20:35:34.801818 14554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 298) took 3433 ms (client timeout 1942 ms). Trace:
W20260806 20:35:34.787885 14546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1444) took 3252 ms (client timeout 2745 ms). Trace:
W20260806 20:35:33.563946 16747 log.cc:924] Time spent T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
W20260806 20:35:34.052345 16415 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.347s user 0.058s sys 0.077s
I20260806 20:35:35.091730 14369 catalog_manager.cc:5495] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 3 to 4, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 4 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:35.094928 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 264) took 3546 ms. Trace:
W20260806 20:35:35.088524 14629 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.542s user 0.030s sys 0.068s
W20260806 20:35:35.100754 16745 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.597s user 0.001s sys 0.004s
W20260806 20:35:33.566074 16655 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.167s user 0.025s sys 0.017s
I20260806 20:35:35.104427 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1714) took 3670 ms. Trace:
W20260806 20:35:35.084678 14546 rpcz_store.cc:269] 0806 20:35:31.535305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.787242 (+3251937us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:34.787556 (+ 314us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 20:35:34.197176 15151 connection.cc:232] RPC call timeout handler was delayed by -2.3561662680003792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.269s for a response
W20260806 20:35:34.565817 16473 consensus_meta.cc:287] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.788s user 0.016s sys 0.001s
W20260806 20:35:33.707831 16677 consensus_meta.cc:287] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.178s user 0.009s sys 0.007s
W20260806 20:35:34.364573 16524 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.866s user 0.024s sys 0.021s
W20260806 20:35:34.209262 14547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 294) took 3637 ms (client timeout 2220 ms). Trace:
W20260806 20:35:34.204435 16465 consensus_meta.cc:287] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.147s user 0.008s sys 0.004s
I20260806 20:35:35.114974 16695 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:35.115494 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 266) took 2921 ms. Trace:
W20260806 20:35:35.081838 14555 rpcz_store.cc:269] 0806 20:35:31.019525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.208659 (+1189134us) service_pool.cc:227] Handling call
0806 20:35:35.030949 (+2822290us) spinlock_profiling.cc:243] Waited 2.69 s on lock 0x7b54002742d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:35.031241 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6181958400}
I20260806 20:35:35.098125 14470 rpcz_store.cc:276] 0806 20:35:31.537453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.776887 (+1239434us) service_pool.cc:227] Handling call
0806 20:35:32.813718 (+ 36831us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:32.816635 (+ 2917us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:32.816676 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:32.844259 (+ 27583us) log.cc:842] Serialized 26 byte log entry
0806 20:35:34.049925 (+1205666us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:34.049965 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:34.050038 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:34.759631 (+709593us) raft_consensus.cc:1671] finished
0806 20:35:34.759701 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.084233 (+324532us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2411,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":24798,"thread_start_us":1180001,"threads_started":1,"wal-append.queue_time_us":1738777}
I20260806 20:35:34.202085 16572 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.118454 16695 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.107895 16714 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:34.445327 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1370) took 3118 ms. Trace:
W20260806 20:35:34.482674 16303 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.055s user 0.004s sys 0.019s
W20260806 20:35:34.201915 16525 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.106s user 0.012s sys 0.000s
W20260806 20:35:34.420943 16723 consensus_meta.cc:287] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.433s user 0.013s sys 0.002s
I20260806 20:35:35.121991 16750 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.124657 16750 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.125206 16722 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:34.195974 15821 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:35.090317 15152 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.038s user 0.020s sys 0.040s
I20260806 20:35:34.206131 16761 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:34.600340 16731 consensus_meta.cc:287] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.555s user 0.011s sys 0.004s
I20260806 20:35:33.440920 16623 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:35.118496 16730 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 2.239s user 0.011s sys 0.030s
W20260806 20:35:35.082829 14554 rpcz_store.cc:269] 0806 20:35:31.368598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.891659 (+1523061us) service_pool.cc:227] Handling call
0806 20:35:34.801225 (+1909566us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54000861d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:34.801538 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":741466880}
W20260806 20:35:34.205475 15179 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 371) took 3765 ms (client timeout 3394 ms). Trace:
I20260806 20:35:35.112757 16687 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:33.593922 16732 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:34.685950 16737 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.095996 16742 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:35.124733 16723 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:34.210260 16753 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:35.129767 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1644) took 4652 ms. Trace:
I20260806 20:35:34.075249 16707 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:35.133706 16681 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.125751 16633 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:35.134130 14476 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260806 20:35:35.105479 16705 consensus_meta.cc:287] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.361s user 0.010s sys 0.003s
I20260806 20:35:35.136864 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1365) took 3917 ms. Trace:
I20260806 20:35:34.193986 16746 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:35.148967 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.979s user 0.000s sys 0.016s
I20260806 20:35:34.996862 16657 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:35.082276 14625 rpcz_store.cc:269] 0806 20:35:32.246367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.088068 (+841701us) service_pool.cc:227] Handling call
0806 20:35:34.815015 (+1726947us) spinlock_profiling.cc:243] Waited 62.3 ms on lock 0x7b5400303254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:34.815345 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":143267712}
I20260806 20:35:34.997509 16286 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:34.469230 16683 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:34.197614 14624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1719) took 1821 ms (client timeout 2227 ms). Trace:
I20260806 20:35:34.476063 16760 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:34.996490 16675 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:34.996006 16664 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:34.209800 16460 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:33.642361 16678 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:35.164765 14521 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.003s sys 0.018s
W20260806 20:35:34.216086 14552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1446) took 2679 ms (client timeout 1982 ms). Trace:
I20260806 20:35:34.542271 16715 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:34.217437 16713 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:34.083830 16697 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.116082 14473 rpcz_store.cc:276] 0806 20:35:32.191382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.981560 (+790178us) service_pool.cc:227] Handling call
0806 20:35:32.987514 (+ 5954us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:32.991946 (+ 4432us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:32.991990 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:32.992552 (+ 562us) log.cc:842] Serialized 26 byte log entry
0806 20:35:33.447122 (+454570us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:33.447178 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:33.447286 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:35.096094 (+1648808us) raft_consensus.cc:1671] finished
0806 20:35:35.101806 (+ 5712us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.112909 (+ 11103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3923,"raft.queue_time_us":4845,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"thread_start_us":444863,"threads_started":1,"wal-append.queue_time_us":1543316}
I20260806 20:35:34.413504 16672 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:35.153262 16656 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:35.153776 16681 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:35.153949 16677 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:35.146929 14478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.998s user 0.009s sys 0.030s
I20260806 20:35:35.155817 16705 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.187877 14549 raft_consensus.cc:2379] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:35.187173 16746 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:33.286217 16396 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:35.144613 14553 rpcz_store.cc:269] 0806 20:35:32.413769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.187100 (+1773331us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:34.187255 (+ 155us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 20:35:35.173877 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.019s user 0.004s sys 0.018s
W20260806 20:35:35.157768 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.000s sys 0.015s
I20260806 20:35:35.158142 16654 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:35.163581 14471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.997s user 0.014s sys 0.010s
I20260806 20:35:35.079532 14626 rpcz_store.cc:276] 0806 20:35:31.430499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.430652 (+ 153us) service_pool.cc:227] Handling call
0806 20:35:31.495458 (+ 64806us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:31.542088 (+ 46630us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:31.542118 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:31.542601 (+ 483us) log.cc:842] Serialized 26 byte log entry
0806 20:35:34.059466 (+2516865us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:34.059501 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:34.816793 (+757292us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b54002be8d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:34.816890 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:34.816987 (+ 97us) raft_consensus.cc:1671] finished
0806 20:35:34.817004 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:34.888918 (+ 71914us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":151694,"raft.run_cpu_time_us":14090,"raft.run_wall_time_us":393545,"spinlock_wait_cycles":106199424,"thread_start_us":2443549,"threads_started":1,"wal-append.queue_time_us":1461660}
I20260806 20:35:35.148275 14632 rpcz_store.cc:276] 0806 20:35:33.012864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.064841 (+1051977us) service_pool.cc:227] Handling call
0806 20:35:34.180372 (+115531us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:35.164889 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.085s user 0.001s sys 0.017s
I20260806 20:35:35.151330 16687 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:35.165283 15153 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.006s sys 0.010s
W20260806 20:35:35.177323 16524 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.628s user 0.024s sys 0.043s
W20260806 20:35:35.176267 14521 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.114s user 0.004s sys 0.019s
I20260806 20:35:35.119830 16572 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:35.164062 16727 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:35.151297 16707 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:35.175711 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.042s user 0.008s sys 0.012s
I20260806 20:35:35.166564 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:35.106818 14631 rpcz_store.cc:276] 0806 20:35:31.430858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.436014 (+ 5156us) service_pool.cc:227] Handling call
0806 20:35:31.436404 (+ 390us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:31.481691 (+ 45287us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:31.481744 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:31.519430 (+ 37686us) log.cc:842] Serialized 26 byte log entry
0806 20:35:32.427881 (+908451us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b48001a01d8. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b839a7f1 00007fb9b83918e7 00007fb9b839177a 00007fb9b848595f 00007fb9b8485627 00007fb9b8499b2c 00007fb9b8499ad8 00007fb9b8499a58 00007fb9b8499a13 00007fb9b8498cbc 00007fb9b741abe6
0806 20:35:34.217906 (+1790025us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b18002a1120. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b318282c 00007fb9b05ad9dc 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a411
0806 20:35:34.218108 (+ 202us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:34.218157 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:34.218247 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:35.101147 (+882900us) raft_consensus.cc:1671] finished
0806 20:35:35.101199 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.101496 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12313,"raft.queue_time_us":214616,"raft.run_cpu_time_us":7934,"raft.run_wall_time_us":770640,"spinlock_wait_cycles":25369728,"thread_start_us":1427718,"threads_started":1,"wal-append.queue_time_us":1468626}
I20260806 20:35:35.148088 14479 rpcz_store.cc:276] 0806 20:35:30.476783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.476912 (+ 129us) service_pool.cc:227] Handling call
0806 20:35:30.504800 (+ 27888us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:30.545128 (+ 40328us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:30.545171 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:30.550714 (+ 5543us) log.cc:842] Serialized 52 byte log entry
0806 20:35:31.379351 (+828637us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b1800465460. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a411 00007fb9b34c75cf
0806 20:35:32.997319 (+1617968us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:34.054290 (+1056971us) log.cc:842] Serialized 14 byte log entry
0806 20:35:34.054492 (+ 202us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:34.054615 (+ 123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:35.128934 (+1074319us) raft_consensus.cc:1671] finished
0806 20:35:35.129155 (+ 221us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.129521 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":909387,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":79732,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":125157,"spinlock_wait_cycles":22966016,"thread_start_us":2638574,"threads_started":2,"wal-append.queue_time_us":2542041}
I20260806 20:35:34.204948 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:34.199657 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:35.179282 14555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1457) took 2243 ms (client timeout 1511 ms). Trace:
W20260806 20:35:35.180326 14554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1451) took 3113 ms (client timeout 2731 ms). Trace:
I20260806 20:35:33.525117 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:35.222680 14477 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:35.178866 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1648) took 4522 ms. Trace:
W20260806 20:35:35.216753 14478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.160s user 0.010s sys 0.031s
W20260806 20:35:35.255769 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.201s user 0.008s sys 0.016s
W20260806 20:35:35.254222 14554 rpcz_store.cc:269] 0806 20:35:32.066646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.169128 (+3102482us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:35.180250 (+ 11122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 20:35:35.254889 14555 rpcz_store.cc:269] 0806 20:35:32.931225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.166522 (+2235297us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:35.174401 (+ 7879us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:35:35.187243 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:35.257869 15153 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.075s user 0.010s sys 0.015s
W20260806 20:35:35.257645 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.076s user 0.000s sys 0.019s
W20260806 20:35:35.216562 14551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1453) took 2948 ms (client timeout 2038 ms). Trace:
W20260806 20:35:35.258157 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.178s user 0.006s sys 0.018s
I20260806 20:35:35.263967 16545 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:35.265264 16545 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.266790 14469 raft_consensus.cc:2379] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
W20260806 20:35:35.267489 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1663) took 1982 ms (client timeout 1638 ms). Trace:
W20260806 20:35:35.267946 14469 rpcz_store.cc:269] 0806 20:35:33.285128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.325226 (+ 40098us) service_pool.cc:227] Handling call
0806 20:35:35.267345 (+1942119us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:35.182641 16587 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.000s sys 0.018s
I20260806 20:35:35.220041 16681 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:35.220599 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1655) took 2075 ms (client timeout 2112 ms). Trace:
W20260806 20:35:35.177403 14624 rpcz_store.cc:269] 0806 20:35:32.376195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.191802 (+815607us) service_pool.cc:227] Handling call
0806 20:35:34.197540 (+1005738us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:35.262704 14551 rpcz_store.cc:269] 0806 20:35:32.175217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.187576 (+2012359us) service_pool.cc:227] Handling call
0806 20:35:35.117478 (+929902us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:35.176975 16653 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.401s user 0.045s sys 0.078s
I20260806 20:35:35.285974 16545 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:35.288371 16460 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:35.288662 14552 rpcz_store.cc:269] 0806 20:35:31.536214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.021162 (+1484948us) service_pool.cc:227] Handling call
0806 20:35:34.215838 (+1194676us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54003e94d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:34.215994 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152203008}
W20260806 20:35:35.289211 14549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1450) took 3678 ms (client timeout 2128 ms). Trace:
I20260806 20:35:35.290760 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:35.291976 14470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1661) took 2061 ms (client timeout 2110 ms). Trace:
I20260806 20:35:35.292927 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:35.292426 15179 rpcz_store.cc:269] 0806 20:35:30.440113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.456779 (+ 16666us) service_pool.cc:227] Handling call
0806 20:35:34.205109 (+3748330us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b54005cd854. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:34.205334 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4560720256}
I20260806 20:35:35.216390 16761 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:35.222391 16656 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:35.213382 16705 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.273603 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1647) took 4741 ms. Trace:
W20260806 20:35:35.285197 14471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.224s user 0.014s sys 0.017s
I20260806 20:35:35.255281 14475 rpcz_store.cc:276] 0806 20:35:30.563349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.563829 (+ 480us) service_pool.cc:227] Handling call
0806 20:35:30.565069 (+ 1240us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:30.568785 (+ 3716us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:30.568810 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:32.601193 (+2032383us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b2c0006bf28. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:32.601644 (+ 451us) log.cc:842] Serialized 26 byte log entry
0806 20:35:34.071060 (+1469416us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:34.071119 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:34.071234 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:34.768778 (+697544us) raft_consensus.cc:1671] finished
0806 20:35:34.768822 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.085427 (+316605us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":104201,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":21175,"spinlock_wait_cycles":92994944,"thread_start_us":1469034,"threads_started":1,"wal-append.queue_time_us":1919758}
W20260806 20:35:35.294956 14632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1717) took 2919 ms (client timeout 2891 ms). Trace:
I20260806 20:35:35.379767 16672 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.380220 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:35.381040 14475 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:35.380679 16672 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.385655 14479 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:35.212321 16746 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:35.215062 16695 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.215046 16624 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:35.393864 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1664) took 2062 ms (client timeout 2687 ms). Trace:
W20260806 20:35:35.394917 14479 rpcz_store.cc:269] 0806 20:35:33.331946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.269475 (+1937529us) service_pool.cc:227] Handling call
0806 20:35:35.393802 (+124327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.358183 14473 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:35.396786 14473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1660) took 2216 ms (client timeout 2506 ms). Trace:
I20260806 20:35:35.384238 14476 rpcz_store.cc:276] 0806 20:35:30.532748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.533661 (+ 913us) service_pool.cc:227] Handling call
0806 20:35:30.559833 (+ 26172us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:35.154969 (+4595136us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:35.224588 (+ 69619us) spinlock_profiling.cc:243] Waited 4.57 s on lock 0x7b5400116cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:35.273534 (+ 48946us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10500535168}
I20260806 20:35:35.391233 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:35.391752 14549 rpcz_store.cc:269] 0806 20:35:31.610721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.324659 (+1713938us) service_pool.cc:227] Handling call
0806 20:35:35.288871 (+1964212us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b5400086bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:35.289107 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4233423744}
I20260806 20:35:35.394307 16460 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:35.395301 14553 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:35.378033 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 275) took 4159 ms. Trace:
I20260806 20:35:35.395742 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.386168 16786 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.396929 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:35.388079 16761 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:35.399219 16713 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.401000 16732 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.213754 14442 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:35.400655 14369 catalog_manager.cc:5495] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 3, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 3 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 20:35:35.401836 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1377) took 3741 ms (client timeout 3697 ms). Trace:
I20260806 20:35:35.402334 16396 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.403406 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:35.400115 16683 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:35.215351 16664 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:34.352079 16486 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:35.388935 14470 rpcz_store.cc:269] 0806 20:35:33.230862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.137168 (+1906306us) service_pool.cc:227] Handling call
0806 20:35:35.291869 (+154701us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:35.260326 16655 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.729s user 0.046s sys 0.055s
I20260806 20:35:35.215780 16633 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.214282 16760 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:35.256608 14547 rpcz_store.cc:269] 0806 20:35:30.571956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.105952 (+1533996us) service_pool.cc:227] Handling call
0806 20:35:34.209022 (+2103070us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b5400357354. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:34.209173 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1132786048}
I20260806 20:35:35.156718 14630 rpcz_store.cc:276] 0806 20:35:31.219675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.228574 (+ 8899us) service_pool.cc:227] Handling call
0806 20:35:31.260865 (+ 32291us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:31.534485 (+273620us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:31.534528 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:31.545309 (+ 10781us) log.cc:842] Serialized 52 byte log entry
0806 20:35:33.020631 (+1475322us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:33.020690 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:33.076885 (+ 56195us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b540006f9d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:33.076935 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:35.134139 (+2057204us) raft_consensus.cc:1671] finished
0806 20:35:35.134183 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.136613 (+ 2430us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":200312,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":41982,"spinlock_wait_cycles":335958912,"thread_start_us":1513752,"threads_started":2,"wal-append.queue_time_us":1416917}
W20260806 20:35:35.687882 14626 rpcz_store.cc:269] 0806 20:35:31.660715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.401598 (+3740883us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:35.401727 (+ 129us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:35:35.651830 14633 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:35.622030 14475 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:35.217690 16696 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:35.294816 14474 rpcz_store.cc:269] 0806 20:35:33.062607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.186916 (+1124309us) service_pool.cc:227] Handling call
0806 20:35:35.131327 (+944411us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:35.678640 16286 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.500s user 0.006s sys 0.007s
I20260806 20:35:35.693840 14627 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:35.694753 14552 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 2.
I20260806 20:35:35.210330 16658 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:35.704828 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:35.260732 16657 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:35.375636 15150 connection.cc:232] RPC call timeout handler was delayed by -1.1702185889880639s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
W20260806 20:35:35.687299 14473 rpcz_store.cc:269] 0806 20:35:33.180226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.186720 (+2006494us) service_pool.cc:227] Handling call
0806 20:35:35.396728 (+210008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.214587 16790 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:35.295490 14477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1653) took 2233 ms (client timeout 2227 ms). Trace:
W20260806 20:35:35.295773 15151 connection.cc:232] RPC call timeout handler was delayed by -1.8566698330105282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2177s for a response
W20260806 20:35:35.379801 14632 rpcz_store.cc:269] 0806 20:35:32.375482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.294686 (+2919204us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:35.294862 (+ 176us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 20:35:35.706347 14553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1460) took 1625 ms (client timeout 1957 ms). Trace:
I20260806 20:35:35.695788 16760 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:35.715165 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1380) took 3469 ms (client timeout 2731 ms). Trace:
I20260806 20:35:35.215994 16753 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:35.347537 16587 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s user 0.001s sys 0.037s
I20260806 20:35:35.729315 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.855669 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.363157 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.788862 14631 raft_consensus.cc:1261] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:35.690281 16672 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:35.700312 16732 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.784765 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 267) took 3592 ms. Trace:
I20260806 20:35:35.784557 14550 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:35.862576 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:35.863303 16732 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:35.712921 14552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1582) took 2374 ms (client timeout 3029 ms). Trace:
W20260806 20:35:35.864023 14552 rpcz_store.cc:269] 0806 20:35:33.338555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.390805 (+2052250us) service_pool.cc:227] Handling call
0806 20:35:35.712837 (+322032us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.864115 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.865097 14547 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:35.865273 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.866052 16486 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:35.324389 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 1.21608e+06us
W20260806 20:35:35.652951 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 277659us
W20260806 20:35:35.828507 15151 connection.cc:232] RPC call timeout handler was delayed by -1.6696457299985923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
W20260806 20:35:35.828877 14553 rpcz_store.cc:269] 0806 20:35:34.061855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.290103 (+1228248us) service_pool.cc:227] Handling call
0806 20:35:35.686826 (+396723us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.785383 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:35.783994 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:35.216938 16743 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:35.851425 14368 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45970 (request call id 233) took 21183 ms (client timeout 15000 ms). Trace:
I20260806 20:35:35.510596 16545 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.220335 16623 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.852783 16658 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:35.853062 14546 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:35.359040 16654 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.853511 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:35.794292 16628 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.664s user 0.004s sys 0.007s
I20260806 20:35:35.477627 16624 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:35.323145 16303 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.715s user 0.004s sys 0.030s
I20260806 20:35:35.269225 16750 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.793284 14479 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:35.828152 14627 rpcz_store.cc:269] 0806 20:35:32.245479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.068509 (+1823030us) service_pool.cc:227] Handling call
0806 20:35:35.715051 (+1646542us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:35.829070 14477 rpcz_store.cc:269] 0806 20:35:33.062001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.188948 (+1126947us) service_pool.cc:227] Handling call
0806 20:35:35.284312 (+1095364us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.856887 14554 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:35.866164 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.788789 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:35.866271 14552 raft_consensus.cc:2396] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:35:35.377908 16572 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:35.785373 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.878311 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.879170 16790 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.295679 16675 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.881641 16677 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.058133 16797 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:36.060882 14626 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:36.061323 16675 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.062342 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.063051 14626 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:36.063467 16790 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:35.785053 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:36.064703 14630 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:36.065129 14555 raft_consensus.cc:2379] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:36.065707 14555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1439) took 5047 ms (client timeout 4768 ms). Trace:
I20260806 20:35:36.065979 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:36.066386 16791 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.675s user 0.001s sys 0.011s
W20260806 20:35:36.066180 14555 rpcz_store.cc:269] 0806 20:35:31.018684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.284916 (+4266232us) service_pool.cc:227] Handling call
0806 20:35:36.065615 (+780699us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.067376 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:35.687059 14629 rpcz_store.cc:276] 0806 20:35:31.218179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.906973 (+1688794us) service_pool.cc:227] Handling call
0806 20:35:32.908839 (+ 1866us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:33.367554 (+458715us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:33.367583 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:33.367994 (+ 411us) log.cc:842] Serialized 26 byte log entry
0806 20:35:35.149196 (+1781202us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:35.149518 (+ 322us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:35.377376 (+227858us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b5400051754. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:35.377447 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:35.377546 (+ 99us) raft_consensus.cc:1671] finished
0806 20:35:35.377569 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.377912 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28872,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":106442,"spinlock_wait_cycles":1054054656,"thread_start_us":1560127,"threads_started":1,"wal-append.queue_time_us":1802981}
I20260806 20:35:36.069204 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:35.788293 14633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1725) took 2779 ms (client timeout 2110 ms). Trace:
W20260806 20:35:36.070279 14633 rpcz_store.cc:269] 0806 20:35:33.009368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:34.073721 (+1064353us) service_pool.cc:227] Handling call
0806 20:35:35.788080 (+1714359us) spinlock_profiling.cc:243] Waited 91.8 ms on lock 0x7b5400303754. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:35.788224 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211054464}
I20260806 20:35:36.070799 15821 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:36.071648 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.071470 15821 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.477829 16746 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:36.073504 14554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1456) took 3330 ms (client timeout 3048 ms). Trace:
W20260806 20:35:36.073885 14554 rpcz_store.cc:269] 0806 20:35:32.719363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.287938 (+2568575us) service_pool.cc:227] Handling call
0806 20:35:36.049731 (+761793us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.074395 14549 raft_consensus.cc:2379] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:35.928361 14521 connection.cc:232] RPC call timeout handler was delayed by -1.1022213319956791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
W20260806 20:35:35.898317 15151 connection.cc:232] RPC call timeout handler was delayed by -1.6117253869888373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1942s for a response
I20260806 20:35:36.046327 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1372) took 4602 ms. Trace:
W20260806 20:35:36.074996 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1502) took 1293 ms (client timeout 1563 ms). Trace:
I20260806 20:35:36.075201 14631 rpcz_store.cc:276] 0806 20:35:31.443611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.400734 (+3957123us) service_pool.cc:227] Handling call
0806 20:35:35.788427 (+387693us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:35.858356 (+ 69929us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:35.940376 (+ 82020us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400303750. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:36.046219 (+105843us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":890136320}
W20260806 20:35:35.922673 14518 connection.cc:232] RPC call timeout handler was delayed by -1.2129421349964105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
W20260806 20:35:36.075405 14479 rpcz_store.cc:269] 0806 20:35:34.756409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.396475 (+640066us) service_pool.cc:227] Handling call
0806 20:35:36.049619 (+653144us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.076438 14546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1445) took 4518 ms (client timeout 3697 ms). Trace:
W20260806 20:35:36.076920 14546 rpcz_store.cc:269] 0806 20:35:31.535774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.231472 (+3695698us) service_pool.cc:227] Handling call
0806 20:35:35.395505 (+164033us) spinlock_profiling.cc:243] Waited 6.63 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:36.054516 (+659011us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15245184}
I20260806 20:35:35.906337 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.050096 16396 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:36.057013 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:36.072944 15153 connection.cc:232] RPC call timeout handler was delayed by -1.0411503779760096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
I20260806 20:35:35.792146 16707 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.068126 14555 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:35.938652 16794 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.562s user 0.012s sys 0.009s
I20260806 20:35:36.069767 16714 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:36.079239 16696 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.849828 14469 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
W20260806 20:35:36.125008 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1513) took 1361 ms (client timeout 1584 ms). Trace:
W20260806 20:35:36.125532 14469 rpcz_store.cc:269] 0806 20:35:34.763503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.510537 (+747034us) service_pool.cc:227] Handling call
0806 20:35:36.124915 (+614378us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.126696 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.127298 15821 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.127661 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1500) took 1372 ms. Trace:
W20260806 20:35:36.129673 16793 log.cc:924] Time spent T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.749s user 0.000s sys 0.013s
I20260806 20:35:36.132634 16790 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:36.078702 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1461) took 2015 ms. Trace:
I20260806 20:35:36.134425 14549 rpcz_store.cc:276] 0806 20:35:34.062855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.877927 (+1815072us) service_pool.cc:227] Handling call
0806 20:35:36.078640 (+200713us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.784811 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.064041 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.051467 14625 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:35.888598 15150 connection.cc:232] RPC call timeout handler was delayed by -1.152601534005953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20260806 20:35:35.906787 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.072921 16746 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.082280 14476 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:36.150730 14476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1507) took 1389 ms (client timeout 1639 ms). Trace:
W20260806 20:35:36.151254 14476 rpcz_store.cc:269] 0806 20:35:34.761269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.714689 (+953420us) service_pool.cc:227] Handling call
0806 20:35:36.150651 (+435962us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.082595 16460 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:36.083155 14633 raft_consensus.cc:2379] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:36.083719 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1588) took 1150 ms. Trace:
I20260806 20:35:36.051949 16786 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:36.084224 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:36.084942 16396 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:36.085566 16623 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:35.921432 16647 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:35.906070 16657 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.839766 16723 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.786434 16633 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:35.794837 16761 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:36.086858 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1586) took 1202 ms. Trace:
I20260806 20:35:35.715893 14369 catalog_manager.cc:5495] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:35:36.087152 15151 connection.cc:232] RPC call timeout handler was delayed by -1.5904613470192999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
I20260806 20:35:36.088503 14630 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:35.790104 16683 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:35.900187 16286 consensus_queue.cc:1046] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.295s
I20260806 20:35:35.794440 16713 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.070058 16624 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.090135 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1584) took 1207 ms. Trace:
I20260806 20:35:36.073650 16760 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:35.852126 16664 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.140627 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:35.904644 16740 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.005s sys 0.022s
I20260806 20:35:36.147953 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.154001 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:35.862174 14478 rpcz_store.cc:276] 0806 20:35:32.192023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:33.020319 (+828296us) service_pool.cc:227] Handling call
0806 20:35:33.034399 (+ 14080us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:33.034668 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:34.056312 (+1021644us) log.cc:842] Serialized 14 byte log entry
0806 20:35:35.717215 (+1660903us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:35.784259 (+ 67044us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:35.784383 (+ 124us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:35.784498 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:35.784701 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3775,"prepare.queue_time_us":1005753,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":109315,"raft.queue_time_us":20237,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":14687,"thread_start_us":2222720,"threads_started":2,"wal-append.queue_time_us":1048722}
W20260806 20:35:36.429522 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.014s sys 0.067s
I20260806 20:35:36.256069 14632 raft_consensus.cc:2396] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260806 20:35:35.928606 16655 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:36.155979 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1727) took 3115 ms (client timeout 2506 ms). Trace:
I20260806 20:35:35.269436 14628 rpcz_store.cc:276] 0806 20:35:31.327081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:31.344431 (+ 17350us) service_pool.cc:227] Handling call
0806 20:35:31.352503 (+ 8072us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:31.573843 (+221340us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:31.573876 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:31.574372 (+ 496us) log.cc:842] Serialized 26 byte log entry
0806 20:35:32.892048 (+1317676us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:34.050004 (+1157956us) log.cc:842] Serialized 14 byte log entry
0806 20:35:34.050257 (+ 253us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:34.050364 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:34.445015 (+394651us) raft_consensus.cc:1671] finished
0806 20:35:34.445046 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:34.445240 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36303,"prepare.queue_time_us":1626794,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":202022,"raft.run_cpu_time_us":417,"raft.run_wall_time_us":177113,"thread_start_us":2582289,"threads_started":3,"wal-append.queue_time_us":1377348}
W20260806 20:35:36.444306 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 1.2308e+06us
I20260806 20:35:36.170737 16727 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:36.452710 16727 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.358198 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.446569 14555 rpcz_store.cc:276] 0806 20:35:34.884384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.067104 (+1182720us) service_pool.cc:227] Handling call
0806 20:35:36.086805 (+ 19701us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.928326 16705 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:36.446557 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.753s user 0.014s sys 0.067s
I20260806 20:35:36.197624 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.169967 14626 raft_consensus.cc:2396] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260806 20:35:36.166443 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1520) took 1400 ms (client timeout 1694 ms). Trace:
I20260806 20:35:36.446238 14551 rpcz_store.cc:276] 0806 20:35:34.932996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.784996 (+852000us) service_pool.cc:227] Handling call
0806 20:35:36.083658 (+298662us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.451103 14625 rpcz_store.cc:269] 0806 20:35:33.025095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.403000 (+2377905us) service_pool.cc:227] Handling call
0806 20:35:36.140299 (+737299us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.459868 15151 connection.cc:232] RPC call timeout handler was delayed by -1.4924980770156253s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
I20260806 20:35:36.194586 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 286) took 5604 ms. Trace:
I20260806 20:35:36.250841 14474 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:36.128162 14469 rpcz_store.cc:276] 0806 20:35:34.755308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.126316 (+1371008us) service_pool.cc:227] Handling call
0806 20:35:36.127578 (+ 1262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.194964 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.446514 14552 rpcz_store.cc:276] 0806 20:35:34.882737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.864948 (+982211us) service_pool.cc:227] Handling call
0806 20:35:36.090054 (+225106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:35.866638 16687 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:36.458676 14475 rpcz_store.cc:269] 0806 20:35:34.766007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.867133 (+1101126us) service_pool.cc:227] Handling call
0806 20:35:36.166360 (+299227us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.461031 14471 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.006s sys 0.064s
W20260806 20:35:36.480811 14471 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.691s user 0.006s sys 0.064s
I20260806 20:35:36.181555 14470 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:36.462949 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1518) took 1697 ms (client timeout 1765 ms). Trace:
I20260806 20:35:36.168655 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.456115 16658 raft_consensus.cc:3032] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:36.474730 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:36.483201 14470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1516) took 1718 ms (client timeout 1626 ms). Trace:
I20260806 20:35:36.463600 14550 rpcz_store.cc:276] 0806 20:35:30.568300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:30.568632 (+ 332us) service_pool.cc:227] Handling call
0806 20:35:30.568997 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:35.861936 (+5292939us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.153091 (+291155us) spinlock_profiling.cc:243] Waited 5.21 s on lock 0x7b5400355f54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:36.172498 (+ 19407us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11993181952}
I20260806 20:35:36.471531 14629 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:36.169338 15150 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:36.445209 16740 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.161s user 0.017s sys 0.032s
I20260806 20:35:36.472697 14627 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:36.476058 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.170804 16524 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:36.469626 14554 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:36.481714 14474 rpcz_store.cc:269] 0806 20:35:34.765327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.056629 (+1291302us) service_pool.cc:227] Handling call
0806 20:35:36.462751 (+406122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.504738 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.261377 14624 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:36.505895 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.506039 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.506801 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.504500 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.487164 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:36.483560 14470 rpcz_store.cc:269] 0806 20:35:34.764605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.713186 (+948581us) service_pool.cc:227] Handling call
0806 20:35:36.483124 (+769938us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.497596 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1462) took 2314 ms. Trace:
I20260806 20:35:36.144688 16545 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.483868 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.508174 14474 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:36.503944 14476 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:36.512353 14476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1511) took 1749 ms (client timeout 1965 ms). Trace:
W20260806 20:35:36.512897 14476 rpcz_store.cc:269] 0806 20:35:34.762760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.152089 (+1389329us) service_pool.cc:227] Handling call
0806 20:35:36.512294 (+360205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.510118 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 261) took 5467 ms. Trace:
I20260806 20:35:36.517102 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.507090 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:36.194741 16790 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:36.521608 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1503) took 1751 ms (client timeout 2125 ms). Trace:
I20260806 20:35:36.516041 14471 rpcz_store.cc:276] 0806 20:35:31.042496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:32.539837 (+1497341us) service_pool.cc:227] Handling call
0806 20:35:32.540256 (+ 419us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:32.540553 (+ 297us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:34.061179 (+1520626us) log.cc:842] Serialized 14 byte log entry
0806 20:35:36.505770 (+2444591us) log.cc:842] Serialized 14 byte log entry
0806 20:35:36.509603 (+ 3833us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.509674 (+ 71us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:36.509764 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.509870 (+ 106us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.510060 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1166090,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":5338,"raft.queue_time_us":78377,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":137,"spinlock_wait_cycles":123584512,"thread_start_us":3789436,"threads_started":3,"wal-append.queue_time_us":1316935}
I20260806 20:35:36.251127 16750 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.273s
W20260806 20:35:36.524989 14474 rpcz_store.cc:269] 0806 20:35:34.759617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.504207 (+1744590us) service_pool.cc:227] Handling call
0806 20:35:36.511237 (+ 7030us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.528436 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.528734 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.529220 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.469225 16460 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:36.529923 14474 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:36.531359 14625 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:36.530671 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1508) took 1769 ms. Trace:
I20260806 20:35:36.522148 14479 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
W20260806 20:35:36.532873 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1512) took 1769 ms (client timeout 1801 ms). Trace:
W20260806 20:35:36.533480 14479 rpcz_store.cc:269] 0806 20:35:34.763120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.087773 (+1324653us) service_pool.cc:227] Handling call
0806 20:35:36.475877 (+388104us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:36.532809 (+ 56932us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8323328}
I20260806 20:35:36.532613 14474 rpcz_store.cc:276] 0806 20:35:34.761624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.528957 (+1767333us) service_pool.cc:227] Handling call
0806 20:35:36.530614 (+ 1657us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.521761 14631 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:36.525871 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.526165 16683 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:36.529327 14628 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:36.480398 16623 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:36.526391 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1429) took 6631 ms. Trace:
I20260806 20:35:36.544668 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1589) took 1577 ms. Trace:
I20260806 20:35:36.545140 14551 rpcz_store.cc:276] 0806 20:35:34.946391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.470771 (+1524380us) service_pool.cc:227] Handling call
0806 20:35:36.523482 (+ 52711us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.518545 14549 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:36.514226 14442 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:36.514750 14473 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:36.519116 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.488217 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:36.550936 14478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1504) took 1771 ms (client timeout 1988 ms). Trace:
I20260806 20:35:36.508999 14547 rpcz_store.cc:276] 0806 20:35:34.183450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.089596 (+1906146us) service_pool.cc:227] Handling call
0806 20:35:36.497508 (+407912us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.551375 14478 rpcz_store.cc:269] 0806 20:35:34.760080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.481235 (+1721155us) service_pool.cc:227] Handling call
0806 20:35:36.531822 (+ 50587us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.554875 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1728) took 1060 ms. Trace:
I20260806 20:35:36.554912 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.555320 14625 rpcz_store.cc:276] 0806 20:35:35.482337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.471013 (+988676us) service_pool.cc:227] Handling call
0806 20:35:36.542997 (+ 71984us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.482688 16658 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:36.559530 14633 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:36.546929 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.505896 16746 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:36.560817 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 268) took 1337 ms (client timeout 1596 ms). Trace:
W20260806 20:35:36.569252 14475 rpcz_store.cc:269] 0806 20:35:35.212636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.480550 (+1267914us) service_pool.cc:227] Handling call
0806 20:35:36.549917 (+ 69367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.561280 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1521) took 1784 ms. Trace:
I20260806 20:35:36.559705 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:36.570675 14627 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:36.560160 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 92818.3us
I20260806 20:35:36.543773 14548 rpcz_store.cc:276] 0806 20:35:29.879408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:29.891457 (+ 12049us) service_pool.cc:227] Handling call
0806 20:35:29.891766 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.170221 (+6278455us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.170283 (+ 62us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.170318 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:36.170369 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.470105 (+299736us) spinlock_profiling.cc:243] Waited 6.26 s on lock 0x7b540018e2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:36.470155 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.510472 (+ 40317us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1282,"raft.queue_time_us":2626,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":68762,"spinlock_wait_cycles":14405580672}
I20260806 20:35:36.570362 14471 rpcz_store.cc:276] 0806 20:35:34.766468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.525575 (+1759107us) service_pool.cc:227] Handling call
0806 20:35:36.550702 (+ 25127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.567746 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.574942 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.575481 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.574960 14552 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:36.521246 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1384) took 1119 ms. Trace:
I20260806 20:35:36.577232 14552 raft_consensus.cc:2396] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:35:36.567219 14630 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 5
I20260806 20:35:36.574368 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:36.579319 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 135051us
I20260806 20:35:36.574476 14477 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:36.583365 14477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1515) took 1819 ms (client timeout 1787 ms). Trace:
W20260806 20:35:36.583824 14477 rpcz_store.cc:269] 0806 20:35:34.764193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.074618 (+1310425us) service_pool.cc:227] Handling call
0806 20:35:36.583277 (+508659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.575774 14626 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:36.590408 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1597) took 1379 ms. Trace:
I20260806 20:35:36.590884 14552 rpcz_store.cc:276] 0806 20:35:35.211305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.475261 (+1263956us) service_pool.cc:227] Handling call
0806 20:35:36.590330 (+115069us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.579031 14632 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:36.592545 16796 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.091s user 0.022s sys 0.101s
I20260806 20:35:36.595911 14549 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:36.597589 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.598434 14478 raft_consensus.cc:2379] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:36.054179 14368 rpcz_store.cc:269] 0806 20:35:14.667998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:14.668254 (+ 256us) service_pool.cc:227] Handling call
0806 20:35:15.531951 (+863697us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b5400002dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b944dd6c 00007fb9b9547c3e 00007fb9b7859fb4 00007fb9b7859f42 00007fb9b7859e71 00007fb9b7859dec 00007fb9b78590a2 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:35.851032 (+20319081us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:29.960711 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:30.274061 (+313350us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:30.274096 (+ 35us) write_op.cc:435] Acquired schema lock
0806 20:35:30.274113 (+ 17us) tablet.cc:660] Decoding operations
0806 20:35:30.274711 (+ 598us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:30.274768 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:30.274798 (+ 30us) tablet.cc:683] Acquiring locks for 3 operations
0806 20:35:30.275112 (+ 314us) tablet.cc:699] Row locks acquired
0806 20:35:30.275131 (+ 19us) write_op.cc:260] PREPARE: finished
0806 20:35:30.275277 (+ 146us) write_op.cc:270] Start()
0806 20:35:30.275383 (+ 106us) write_op.cc:276] Timestamp: P: 1786048530275232 usec, L: 0
0806 20:35:30.275402 (+ 19us) op_driver.cc:331] REPLICATION: starting
0806 20:35:30.312442 (+ 37040us) log.cc:842] Serialized 1380 byte log entry
0806 20:35:32.650393 (+2337951us) op_driver.cc:444] REPLICATION: finished
0806 20:35:34.571040 (+1920647us) write_op.cc:301] APPLY: starting
0806 20:35:34.571154 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:34.571367 (+ 213us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:34.584132 (+ 12765us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:34.584745 (+ 613us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:34.585018 (+ 273us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 20:35:34.585083 (+ 65us) write_op.cc:312] APPLY: finished
0806 20:35:34.585364 (+ 281us) log.cc:842] Serialized 35 byte log entry
0806 20:35:34.684973 (+ 99609us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:34.685109 (+ 136us) write_op.cc:454] Released schema lock
0806 20:35:34.685167 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:34.685522 (+ 355us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:34.685568 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1076986,"reported_tablets":17,"spinlock_wait_cycles":1070439680,"child_traces":[["op",{"apply.queue_time_us":1834833,"apply.run_cpu_time_us":8008,"apply.run_wall_time_us":533962,"num_ops":3,"prepare.queue_time_us":253845,"prepare.run_cpu_time_us":38203,"prepare.run_wall_time_us":656288,"raft.queue_time_us":1243445,"raft.run_cpu_time_us":51462,"raft.run_wall_time_us":2568332,"replication_time_us":2367227,"spinlock_wait_cycles":540479232,"thread_start_us":2939905,"threads_started":5,"wal-append.queue_time_us":1646764,"wal-append.run_cpu_time_us":37126,"wal-append.run_wall_time_us":1649793}]]}
I20260806 20:35:36.558149 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.599614 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:36.603216 14547 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:36.602365 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.602245 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:36.603704 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.523633 16761 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.606009 14633 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:36.544216 16760 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.614239 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.614996 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:36.599320 16796 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.123s user 0.022s sys 0.101s
I20260806 20:35:36.626657 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.616093 14628 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:36.611636 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 442432us
I20260806 20:35:36.616797 14631 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:36.531008 16705 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:36.623714 16792 log.cc:924] Time spent T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Append to log took a long time: real 0.944s user 0.000s sys 0.021s
I20260806 20:35:36.623088 14625 raft_consensus.cc:2362] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5. Re-sending same reply.
I20260806 20:35:36.574074 16714 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:36.595162 16727 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.630775 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.631872 14477 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:36.629010 14475 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:36.592985 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.642238 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.532824 16655 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:36.644047 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.638934 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1514) took 1875 ms. Trace:
I20260806 20:35:36.651894 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.592962 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.632933 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.648027 14479 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:36.579696 14624 rpcz_store.cc:276] 0806 20:35:35.401894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:35.784385 (+382491us) service_pool.cc:227] Handling call
0806 20:35:36.518387 (+734002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.592929 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.652103 14475 rpcz_store.cc:276] 0806 20:35:34.763844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.570153 (+1806309us) service_pool.cc:227] Handling call
0806 20:35:36.638867 (+ 68714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.652716 14631 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:36.653340 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.651489 14469 raft_consensus.cc:2379] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:36.656349 14546 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:36.620115 16633 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:36.670879 14626 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:36.658600 14548 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:35:36.624708 16460 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.663035 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.662065 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.618386 14369 catalog_manager.cc:5495] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 2 to 3, leader changed from 148bf2a11dba423ab364371f0458e871 (127.12.248.132) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 3 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:36.634687 16657 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.666285 14551 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:36.599160 16524 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:36.664722 14633 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:36.607003 16623 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.672708 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.673441 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1585) took 1789 ms. Trace:
I20260806 20:35:36.609517 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.674280 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.679143 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.693185 14626 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:36.680091 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1587) took 1795 ms. Trace:
I20260806 20:35:36.693966 14551 rpcz_store.cc:276] 0806 20:35:34.885226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.563212 (+1677986us) service_pool.cc:227] Handling call
0806 20:35:36.680013 (+116801us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.694466 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.681890 16655 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:36.683913 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1505) took 1923 ms. Trace:
I20260806 20:35:36.689509 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 292) took 6107 ms. Trace:
I20260806 20:35:36.699079 14471 rpcz_store.cc:276] 0806 20:35:34.760519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.650560 (+1890041us) service_pool.cc:227] Handling call
0806 20:35:36.683829 (+ 33269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.651485 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.686673 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.690387 14552 rpcz_store.cc:276] 0806 20:35:34.883892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.608953 (+1725061us) service_pool.cc:227] Handling call
0806 20:35:36.673358 (+ 64405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.667649 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.699380 14548 rpcz_store.cc:276] 0806 20:35:30.571418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.645180 (+6073762us) service_pool.cc:227] Handling call
0806 20:35:36.654007 (+ 8827us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.672096 (+ 18089us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.678713 (+ 6617us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.693768 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.701867 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.676402 16633 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:36.716524 14555 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:36.718724 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.719481 14626 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:36.719868 14625 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:36.718184 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.721823 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:36.698746 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1510) took 1930 ms (client timeout 2076 ms). Trace:
W20260806 20:35:36.723148 14469 rpcz_store.cc:269] 0806 20:35:34.762405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.505561 (+1743156us) service_pool.cc:227] Handling call
0806 20:35:36.692643 (+187082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.724161 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.724776 14469 raft_consensus.cc:1261] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:36.725492 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1654) took 3663 ms. Trace:
I20260806 20:35:36.726342 14471 raft_consensus.cc:2396] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:35:36.692266 16286 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:36.729210 14478 raft_consensus.cc:2396] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:35:36.731235 14631 raft_consensus.cc:2379] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:36.736915 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:36.737861 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.714671 16790 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:36.738464 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.765231 14628 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:36.739542 16713 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:36.705632 15183 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
W20260806 20:35:36.783855 16785 log.cc:924] Time spent T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.405s user 0.023s sys 0.004s
I20260806 20:35:36.759382 15184 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:36.789237 15184 raft_consensus.cc:2396] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:35:36.692605 15179 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:36.679244 16714 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:36.790375 14549 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:36.791297 14549 raft_consensus.cc:595] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:36.791738 14549 raft_consensus.cc:603] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:36.792659 14632 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:36.792994 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 65025.9us
I20260806 20:35:36.793774 14550 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:36.794394 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:36.794579 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 288) took 6224 ms. Trace:
I20260806 20:35:36.795092 14632 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:36.795059 14550 rpcz_store.cc:276] 0806 20:35:30.570303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.793133 (+6222830us) service_pool.cc:227] Handling call
0806 20:35:36.793549 (+ 416us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.794269 (+ 720us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.794493 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.795504 14442 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:36.792627 16714 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:36.711615 16633 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:36.797192 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:36.797966 14476 raft_consensus.cc:2362] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2. Re-sending same reply.
I20260806 20:35:36.798517 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 269) took 1585 ms. Trace:
I20260806 20:35:36.798877 14476 rpcz_store.cc:276] 0806 20:35:35.213075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.514040 (+1300965us) service_pool.cc:227] Handling call
0806 20:35:36.797060 (+283020us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593bce 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:36.798459 (+ 1399us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6462208}
I20260806 20:35:36.800618 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:36.800909 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:36.801352 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 73055.4us
I20260806 20:35:36.801857 15181 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:36.802587 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1394) took 1012 ms. Trace:
I20260806 20:35:36.802819 16460 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:36.803125 14625 rpcz_store.cc:276] 0806 20:35:35.790575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.649896 (+859321us) service_pool.cc:227] Handling call
0806 20:35:36.802376 (+152480us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b540020e754. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:36.802517 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68121472}
I20260806 20:35:36.804524 16460 consensus_queue.cc:1046] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:36.771976 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.806277 14633 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:36.772343 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:36.807687 14553 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:36.807819 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:36.775064 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1733) took 1047 ms. Trace:
I20260806 20:35:36.781385 14548 raft_consensus.cc:1226] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
W20260806 20:35:36.808930 14470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1506) took 2048 ms (client timeout 2182 ms). Trace:
I20260806 20:35:36.809471 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1435) took 5794 ms. Trace:
I20260806 20:35:36.809841 14548 rpcz_store.cc:276] 0806 20:35:31.015256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.714234 (+5698978us) service_pool.cc:227] Handling call
0806 20:35:36.776444 (+ 62210us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.809217 (+ 32773us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.809400 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.801424 14547 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:36.808179 14553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1583) took 1926 ms (client timeout 2125 ms). Trace:
I20260806 20:35:36.777431 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1590) took 1830 ms. Trace:
I20260806 20:35:36.811560 14546 rpcz_store.cc:276] 0806 20:35:34.947418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.077818 (+1130400us) service_pool.cc:227] Handling call
0806 20:35:36.474444 (+396626us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:36.777246 (+302802us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b540018ddd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:36.777362 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":571413632}
W20260806 20:35:36.809478 14470 rpcz_store.cc:269] 0806 20:35:34.760909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.508891 (+1747982us) service_pool.cc:227] Handling call
0806 20:35:36.807627 (+298736us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:36.808862 (+ 1235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2920832}
I20260806 20:35:36.730182 16286 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:36.808682 14629 rpcz_store.cc:276] 0806 20:35:35.727571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.068810 (+341239us) service_pool.cc:227] Handling call
0806 20:35:36.774826 (+706016us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b5400180bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:36.774970 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":923683072}
W20260806 20:35:36.811327 14553 rpcz_store.cc:269] 0806 20:35:34.882124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.075873 (+1193749us) service_pool.cc:227] Handling call
0806 20:35:36.772210 (+696337us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593bce 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:36.808121 (+ 35911us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8142208}
I20260806 20:35:36.770490 14473 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
W20260806 20:35:36.821198 14473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1519) took 2055 ms (client timeout 1952 ms). Trace:
W20260806 20:35:36.821659 14473 rpcz_store.cc:269] 0806 20:35:34.765655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.083906 (+1318251us) service_pool.cc:227] Handling call
0806 20:35:36.821107 (+737201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.792280 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48150 (request call id 18) took 6049 ms. Trace:
I20260806 20:35:36.824528 14549 rpcz_store.cc:276] 0806 20:35:30.742897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.790078 (+6047181us) service_pool.cc:227] Handling call
0806 20:35:36.792175 (+ 2097us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.817443 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:36.817329 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 634256us
W20260806 20:35:36.829203 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 226314us
I20260806 20:35:36.743112 16396 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:36.777877 16525 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.117s user 0.058s sys 0.048s
I20260806 20:35:36.725821 14469 rpcz_store.cc:276] 0806 20:35:33.062318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.723960 (+3661642us) service_pool.cc:227] Handling call
0806 20:35:36.724384 (+ 424us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.725207 (+ 823us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.725401 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.843798 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:36.844811 14628 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:36.845379 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1433) took 5821 ms. Trace:
I20260806 20:35:36.816184 14475 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:36.810855 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.755970 16753 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.743472 16624 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:36.835039 14625 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [3.2-5.4]
I20260806 20:35:36.845942 14551 rpcz_store.cc:276] 0806 20:35:31.013980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.712935 (+5698955us) service_pool.cc:227] Handling call
0806 20:35:36.777688 (+ 64753us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.835335 (+ 57647us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.835354 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.835383 (+ 29us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:36.835407 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.835453 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.835532 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.854140 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1434) took 5839 ms. Trace:
I20260806 20:35:36.854725 14552 rpcz_store.cc:276] 0806 20:35:31.014607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.722540 (+5707933us) service_pool.cc:227] Handling call
0806 20:35:36.826172 (+103632us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.841599 (+ 15427us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.841641 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.853792 (+ 12151us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:36.853851 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.853915 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.854044 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.828609 16286 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:36.860618 14479 raft_consensus.cc:1261] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260806 20:35:36.835807 16525 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.353s user 0.058s sys 0.049s
I20260806 20:35:36.779299 16743 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.864056 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1374) took 5235 ms. Trace:
I20260806 20:35:36.864567 14625 rpcz_store.cc:276] 0806 20:35:31.616785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.804224 (+5187439us) service_pool.cc:227] Handling call
0806 20:35:36.819820 (+ 15596us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:35:36.851712 (+ 31892us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.852006 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.866654 16761 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:36.866734 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.867839 14551 raft_consensus.cc:1226] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
W20260806 20:35:36.868266 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 72865.8us
I20260806 20:35:36.868528 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1452) took 4800 ms. Trace:
I20260806 20:35:36.868925 14551 rpcz_store.cc:276] 0806 20:35:32.067811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.862030 (+4794219us) service_pool.cc:227] Handling call
0806 20:35:36.867668 (+ 5638us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.868275 (+ 607us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.868439 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.869117 14442 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:36.866986 14549 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:35:36.865808 14550 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:36.871372 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54486 (request call id 1224) took 3653 ms. Trace:
I20260806 20:35:36.882983 14551 rpcz_store.cc:276] 0806 20:35:33.218483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.869837 (+3651354us) service_pool.cc:227] Handling call
0806 20:35:36.871271 (+ 1434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.875142 14626 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:36.884222 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1436) took 5868 ms. Trace:
I20260806 20:35:36.878711 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1447) took 5335 ms. Trace:
I20260806 20:35:36.884672 14550 rpcz_store.cc:276] 0806 20:35:31.015981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.796302 (+5780321us) service_pool.cc:227] Handling call
0806 20:35:36.831646 (+ 35344us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:36.883957 (+ 52311us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.884171 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.885098 14549 rpcz_store.cc:276] 0806 20:35:31.536517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.861757 (+5325240us) service_pool.cc:227] Handling call
0806 20:35:36.866667 (+ 4910us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.871960 (+ 5293us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.872127 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.877844 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1652) took 3804 ms. Trace:
I20260806 20:35:36.887190 14479 rpcz_store.cc:276] 0806 20:35:33.059377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.732216 (+3672839us) service_pool.cc:227] Handling call
0806 20:35:36.840976 (+108760us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.863595 (+ 22619us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.863936 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.879522 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1437) took 5862 ms. Trace:
I20260806 20:35:36.888645 14548 rpcz_store.cc:276] 0806 20:35:31.017034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.816747 (+5799713us) service_pool.cc:227] Handling call
0806 20:35:36.833098 (+ 16351us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.871771 (+ 38673us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.871796 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.879168 (+ 7372us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:36.879243 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.879323 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.879456 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.872604 14631 raft_consensus.cc:2396] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:36.890033 14627 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:36.893545 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1459) took 2825 ms. Trace:
I20260806 20:35:36.893682 16664 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:36.894255 16664 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:36.894109 14549 rpcz_store.cc:276] 0806 20:35:34.061246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.885951 (+2824705us) service_pool.cc:227] Handling call
0806 20:35:36.886344 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.886542 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.886564 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.886603 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:36.886650 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.886700 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.886821 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.896344 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 536741us
I20260806 20:35:36.896481 14549 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:36.896780 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.898109 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.898288 14549 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:36.898794 15178 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:36.899056 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.899659 15186 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:36.899786 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:36.900879 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:36.901578 15179 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:36.903313 14633 raft_consensus.cc:1226] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 2. Current term is 3. Ops: []
I20260806 20:35:36.904105 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1441) took 5860 ms. Trace:
I20260806 20:35:36.904600 14553 rpcz_store.cc:276] 0806 20:35:31.043544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.840584 (+5797040us) service_pool.cc:227] Handling call
0806 20:35:36.889648 (+ 49064us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.903764 (+ 14116us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.903794 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.903833 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:36.903876 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.903940 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.904044 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.904280 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1659) took 3789 ms. Trace:
I20260806 20:35:36.894763 16664 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:36.911116 14369 catalog_manager.cc:5495] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 4, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 4 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:36.913625 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1395) took 1122 ms. Trace:
I20260806 20:35:36.914007 14631 rpcz_store.cc:276] 0806 20:35:35.790904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.765072 (+974168us) service_pool.cc:227] Handling call
0806 20:35:36.913559 (+148487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.907687 14477 rpcz_store.cc:276] 0806 20:35:33.099066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.773333 (+3674267us) service_pool.cc:227] Handling call
0806 20:35:36.800991 (+ 27658us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.862282 (+ 61291us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.862328 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.877412 (+ 15084us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:36.887977 (+ 10565us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.888059 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.888164 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31786,"raft.queue_time_us":74350,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":7810}
I20260806 20:35:36.915169 14629 raft_consensus.cc:1261] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:36.910183 14546 raft_consensus.cc:1261] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:36.919472 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1438) took 5901 ms. Trace:
I20260806 20:35:36.913983 15151 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:36.919938 14546 rpcz_store.cc:276] 0806 20:35:31.018017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.828883 (+5810866us) service_pool.cc:227] Handling call
0806 20:35:36.874893 (+ 46010us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.919183 (+ 44290us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.919406 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.873369 16396 raft_consensus.cc:3032] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:36.921972 16396 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:36.922912 16396 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:36.924692 16545 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:36.925256 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 62660.2us
I20260806 20:35:36.926688 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1718) took 4540 ms. Trace:
I20260806 20:35:36.922236 14369 catalog_manager.cc:5495] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 4 to 5, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 5 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:36.927107 14629 rpcz_store.cc:276] 0806 20:35:32.375864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.840557 (+4464693us) service_pool.cc:227] Handling call
0806 20:35:36.892608 (+ 52051us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.915625 (+ 23017us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.915863 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.886835 16657 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:36.813126 16658 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:36.938508 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1448) took 5391 ms. Trace:
W20260806 20:35:36.940804 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 495085us
I20260806 20:35:36.942498 14627 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:36.941401 14595 connection.cc:232] RPC call timeout handler was delayed by -2.6336645580013283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1988s for a response
I20260806 20:35:36.949818 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1722) took 4572 ms. Trace:
I20260806 20:35:36.950289 14627 rpcz_store.cc:276] 0806 20:35:32.377198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.904729 (+4527531us) service_pool.cc:227] Handling call
0806 20:35:36.933957 (+ 29228us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.949498 (+ 15541us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.949749 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.953910 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 514106us
W20260806 20:35:36.954293 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 91786.4us
I20260806 20:35:36.927222 16705 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:36.965593 14595 connection.cc:232] RPC call timeout handler was delayed by -2.533969089970924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21250000000000002s for a response
I20260806 20:35:36.965840 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1721) took 4589 ms. Trace:
W20260806 20:35:36.966045 14595 connection.cc:232] RPC call timeout handler was delayed by -2.533969089970924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21250000000000002s for a response
W20260806 20:35:36.966524 14595 connection.cc:232] RPC call timeout handler was delayed by -2.4685842079925351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
I20260806 20:35:36.966282 14625 rpcz_store.cc:276] 0806 20:35:32.376892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.878180 (+4501288us) service_pool.cc:227] Handling call
0806 20:35:36.934309 (+ 56129us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.961141 (+ 26832us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.961166 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.965564 (+ 4398us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:36.965624 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.965683 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.965784 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:36.967108 14595 connection.cc:232] RPC call timeout handler was delayed by -2.0671651820011903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
W20260806 20:35:36.967536 14595 connection.cc:232] RPC call timeout handler was delayed by -2.0502213309810031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260806 20:35:36.967862 14595 connection.cc:232] RPC call timeout handler was delayed by -2.0502213309810031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260806 20:35:36.968184 14595 connection.cc:232] RPC call timeout handler was delayed by -1.9706171609868761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
W20260806 20:35:36.969067 14595 connection.cc:232] RPC call timeout handler was delayed by -1.9706171609868761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
W20260806 20:35:36.969453 14595 connection.cc:232] RPC call timeout handler was delayed by -1.9636894309951458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3029s for a response
W20260806 20:35:36.969887 14595 connection.cc:232] RPC call timeout handler was delayed by -1.4305810149817262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1519s for a response
W20260806 20:35:36.970244 14595 connection.cc:232] RPC call timeout handler was delayed by -1.372081014997093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1584s for a response
W20260806 20:35:36.970705 14595 connection.cc:232] RPC call timeout handler was delayed by -1.334281014977023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260806 20:35:36.971310 14595 connection.cc:232] RPC call timeout handler was delayed by -1.334281014977023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260806 20:35:36.971721 14595 connection.cc:232] RPC call timeout handler was delayed by -1.2978011959930882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2846s for a response
W20260806 20:35:36.972091 14595 connection.cc:232] RPC call timeout handler was delayed by -1.273081014980562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
I20260806 20:35:36.937704 16760 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:36.972534 14595 connection.cc:232] RPC call timeout handler was delayed by -1.273081014980562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260806 20:35:36.972926 14595 connection.cc:232] RPC call timeout handler was delayed by -1.2091810149722733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
W20260806 20:35:36.973268 14595 connection.cc:232] RPC call timeout handler was delayed by -1.1893810149922501s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
W20260806 20:35:36.973709 14595 connection.cc:232] RPC call timeout handler was delayed by -1.1767810149758589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20260806 20:35:36.974067 14595 connection.cc:232] RPC call timeout handler was delayed by -1.1767810149758589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
I20260806 20:35:36.970566 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1379) took 4719 ms. Trace:
W20260806 20:35:36.974406 14595 connection.cc:232] RPC call timeout handler was delayed by -1.0892645579879172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.37040000000000006s for a response
W20260806 20:35:36.974828 14595 connection.cc:232] RPC call timeout handler was delayed by -1.0408810149820056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260806 20:35:36.975051 14548 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:36.974562 14633 rpcz_store.cc:276] 0806 20:35:32.245136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.806875 (+4561739us) service_pool.cc:227] Handling call
0806 20:35:36.890488 (+ 83613us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.908871 (+ 18383us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.964035 (+ 55164us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b5400312250. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:36.964225 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191462656}
I20260806 20:35:36.975891 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1463) took 2772 ms. Trace:
I20260806 20:35:36.976380 14548 rpcz_store.cc:276] 0806 20:35:34.203102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.889463 (+2686361us) service_pool.cc:227] Handling call
0806 20:35:36.955109 (+ 65646us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.975558 (+ 20449us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.975823 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.979982 14628 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:36.983180 14631 raft_consensus.cc:1261] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:36.983942 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1723) took 4600 ms. Trace:
I20260806 20:35:36.984333 14631 rpcz_store.cc:276] 0806 20:35:32.383470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.914641 (+4531171us) service_pool.cc:227] Handling call
0806 20:35:36.964395 (+ 49754us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.983649 (+ 19254us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.983874 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.987712 14627 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 20:35:36.988647 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1458) took 2928 ms. Trace:
I20260806 20:35:36.982174 14596 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:36.982044 14369 catalog_manager.cc:5495] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:36.991011 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.992321 14628 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:36.988029 14627 pending_rounds.cc:76] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 1
I20260806 20:35:36.993934 14625 raft_consensus.cc:1226] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:36.995648 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:36.975842 16623 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:36.980432 16524 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:36.993355 14627 pending_rounds.cc:97] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:35:36.997510 14631 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:36.997327 14518 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:36.978370 16654 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:36.996627 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:37.003589 14633 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:37.000160 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1386) took 1591 ms. Trace:
W20260806 20:35:37.003820 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 201591us
I20260806 20:35:37.004073 14625 rpcz_store.cc:276] 0806 20:35:35.403263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.967459 (+1564196us) service_pool.cc:227] Handling call
0806 20:35:36.989296 (+ 21837us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.994327 (+ 5031us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.994535 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:37.007470 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48538.6us
I20260806 20:35:37.007699 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:37.008411 14627 raft_consensus.cc:2866] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:35:37.015481 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 101561us
I20260806 20:35:36.992929 16753 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:37.024153 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1729) took 1541 ms. Trace:
I20260806 20:35:37.024689 14631 rpcz_store.cc:276] 0806 20:35:35.482946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.002246 (+1519300us) service_pool.cc:227] Handling call
0806 20:35:37.013121 (+ 10875us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.023694 (+ 10573us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:37.023739 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:37.023799 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:37.023846 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.023927 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.024080 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:36.988998 14550 rpcz_store.cc:276] 0806 20:35:34.060590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.910775 (+2850185us) service_pool.cc:227] Handling call
0806 20:35:36.961045 (+ 50270us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.988316 (+ 27271us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.988348 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.988375 (+ 27us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:36.988401 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.988441 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.988591 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5662}
I20260806 20:35:37.016944 16623 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:37.030589 15153 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.034369 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1730) took 1551 ms. Trace:
I20260806 20:35:36.938992 14555 rpcz_store.cc:276] 0806 20:35:31.547514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.842605 (+5295091us) service_pool.cc:227] Handling call
0806 20:35:36.902709 (+ 60104us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:36.938034 (+ 35325us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:36.938080 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:36.938145 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:36.938216 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:36.938300 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:36.938442 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18004,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":8528}
I20260806 20:35:37.021482 16753 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:37.022272 16654 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:37.034785 14625 rpcz_store.cc:276] 0806 20:35:35.483379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.005389 (+1522010us) service_pool.cc:227] Handling call
0806 20:35:37.011020 (+ 5631us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.016928 (+ 5908us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:37.016976 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:37.029410 (+ 12434us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:37.034128 (+ 4718us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.034198 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.034309 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":503,"raft.queue_time_us":677,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":19086}
I20260806 20:35:37.038757 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:37.036628 14628 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:37.041391 14631 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:37.042652 16753 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:37.043324 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1389) took 1314 ms. Trace:
I20260806 20:35:37.043601 14631 rpcz_store.cc:276] 0806 20:35:35.728705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.031249 (+1302544us) service_pool.cc:227] Handling call
0806 20:35:37.037672 (+ 6423us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.043073 (+ 5401us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.043279 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:37.044798 14631 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:37.045336 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 278) took 1147 ms. Trace:
I20260806 20:35:37.045593 14631 rpcz_store.cc:276] 0806 20:35:35.898068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.044259 (+1146191us) service_pool.cc:227] Handling call
0806 20:35:37.044563 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.045155 (+ 592us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.045283 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:37.046594 14631 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 20:35:37.048954 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 75164.5us
I20260806 20:35:37.034029 16623 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.054508 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1385) took 1652 ms. Trace:
I20260806 20:35:37.054932 14627 rpcz_store.cc:276] 0806 20:35:35.402701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.960700 (+1557999us) service_pool.cc:227] Handling call
0806 20:35:36.971654 (+ 10954us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.054261 (+ 82607us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.054437 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:37.055397 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1731) took 1571 ms. Trace:
W20260806 20:35:37.050695 14356 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 500956bd1a3846a3a7b16c51ddecbc44 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 8e0cfd35f2994f91aaae46c25c6d6bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:35:37.061455 14356 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 68523.2us
I20260806 20:35:37.062762 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1455) took 4357 ms. Trace:
I20260806 20:35:37.051393 14369 catalog_manager.cc:5495] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 3, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 3 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:37.063241 14552 rpcz_store.cc:276] 0806 20:35:32.705410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.870756 (+4165346us) service_pool.cc:227] Handling call
0806 20:35:36.871133 (+ 377us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:36.886265 (+ 15132us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:36.886296 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:36.924621 (+ 38325us) log.cc:842] Serialized 26 byte log entry
0806 20:35:37.013004 (+ 88383us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:37.013116 (+ 112us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.013261 (+ 145us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:37.062541 (+ 49280us) raft_consensus.cc:1671] finished
0806 20:35:37.062569 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.062696 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15188,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":7392,"thread_start_us":82794,"threads_started":1,"wal-append.queue_time_us":126215}
W20260806 20:35:37.083714 16647 consensus_meta.cc:287] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.622s user 0.000s sys 0.011s
I20260806 20:35:37.084535 16647 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.084918 16647 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.085264 16647 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.086189 16753 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.057240 16623 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:37.093310 16753 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:37.132584 16746 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:37.133291 16746 raft_consensus.cc:686] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:37.133409 16786 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:37.146642 14630 consensus_meta.cc:287] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.536s user 0.007s sys 0.004s
I20260806 20:35:37.147297 14630 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:37.148083 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 279) took 1249 ms. Trace:
I20260806 20:35:37.148581 14630 rpcz_store.cc:276] 0806 20:35:35.898475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.459201 (+560726us) service_pool.cc:227] Handling call
0806 20:35:37.148000 (+688799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:37.057318 16696 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:37.082206 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:37.271080 16746 consensus_queue.cc:237] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:37.263736 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 365375us
W20260806 20:35:37.156596 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 135669us
I20260806 20:35:37.175390 14596 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.162689 14521 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.175137 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:37.281512 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1390) took 1552 ms. Trace:
W20260806 20:35:37.167601 14554 consensus_meta.cc:287] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.617s user 0.006s sys 0.003s
I20260806 20:35:37.296835 14625 rpcz_store.cc:276] 0806 20:35:35.729149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.052000 (+1322851us) service_pool.cc:227] Handling call
0806 20:35:37.280690 (+228690us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.281120 (+ 430us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:37.281148 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:37.281200 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:37.281241 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.281317 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.281448 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103}
I20260806 20:35:37.296195 14552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5277702c39f453cb1063cf506183484"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
W20260806 20:35:37.299515 15150 connection.cc:232] RPC call timeout handler was delayed by -1.5256233799736947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
I20260806 20:35:37.299774 14552 raft_consensus.cc:595] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:37.300369 14552 raft_consensus.cc:603] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:37.297024 14554 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
W20260806 20:35:37.301074 14554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1596) took 2147 ms (client timeout 1952 ms). Trace:
W20260806 20:35:37.301362 14554 rpcz_store.cc:269] 0806 20:35:35.153415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.074517 (+921102us) service_pool.cc:227] Handling call
0806 20:35:37.301010 (+1226493us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:37.302351 16761 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.302747 16761 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.303066 16761 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.055796 14633 rpcz_store.cc:276] 0806 20:35:35.483763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.006920 (+1523157us) service_pool.cc:227] Handling call
0806 20:35:37.020508 (+ 13588us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.049235 (+ 28727us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:37.049266 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:37.055158 (+ 5892us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:37.055196 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.055240 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.055323 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16519}
I20260806 20:35:37.306427 14633 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:37.307682 14633 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260806 20:35:37.307796 14441 leader_election.cc:334] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.789s (SENT)
I20260806 20:35:37.310634 16657 consensus_queue.cc:1046] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:35:37.311189 16696 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:37.311406 16657 consensus_queue.cc:1225] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer 8e0cfd35f2994f91aaae46c25c6d6bff log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 20:35:37.311630 16696 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.312031 16696 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:37.313398 16286 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.314257 14444 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:37.315500 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.315771 16286 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:37.316344 14444 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:37.317119 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 219376us
I20260806 20:35:37.317929 14518 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:37.318861 14633 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:37.320295 15150 leader_election.cc:334] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 2.220s (SENT)
I20260806 20:35:37.321166 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1396) took 1529 ms. Trace:
I20260806 20:35:37.321333 16705 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:37.321631 14624 rpcz_store.cc:276] 0806 20:35:35.791280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.035388 (+1244108us) service_pool.cc:227] Handling call
0806 20:35:37.321095 (+285707us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816}
I20260806 20:35:37.134120 15185 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:37.153285 16683 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:37.325017 16683 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.326897 16664 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:37.329026 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 221104us
W20260806 20:35:37.332779 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 293942us
W20260806 20:35:37.263729 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 195473us
I20260806 20:35:37.330304 14626 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:37.334458 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 280) took 1435 ms. Trace:
I20260806 20:35:37.335147 14626 rpcz_store.cc:276] 0806 20:35:35.898795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.727157 (+828362us) service_pool.cc:227] Handling call
0806 20:35:37.334380 (+607223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:37.338598 14595 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:37.226756 16396 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.291471 16658 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:37.328419 16683 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:37.360172 14441 leader_election.cc:334] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.849s (SENT)
W20260806 20:35:37.382918 14632 consensus_meta.cc:287] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.587s user 0.013s sys 0.000s
I20260806 20:35:37.393290 14632 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
W20260806 20:35:37.393978 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 52204.2us
I20260806 20:35:37.198995 14369 catalog_manager.cc:5495] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 3 to 4, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 4 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 20:35:37.382921 14547 consensus_meta.cc:287] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.540s user 0.008s sys 0.001s
I20260806 20:35:37.394963 14547 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:37.395493 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 304) took 1272 ms. Trace:
W20260806 20:35:37.385094 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 302374us
I20260806 20:35:37.395843 14547 rpcz_store.cc:276] 0806 20:35:36.123346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.800278 (+676932us) service_pool.cc:227] Handling call
0806 20:35:37.395427 (+595149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:37.396107 14518 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:37.407891 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 67572.2us
W20260806 20:35:37.400035 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 369560us
W20260806 20:35:37.420701 16633 consensus_meta.cc:287] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.623s user 0.009s sys 0.003s
I20260806 20:35:37.421335 16633 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:37.424026 16633 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:37.425060 14474 raft_consensus.cc:2379] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:37.428278 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1509) took 2666 ms (client timeout 3170 ms). Trace:
W20260806 20:35:37.428757 14474 rpcz_store.cc:269] 0806 20:35:34.762016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.536335 (+1774319us) service_pool.cc:227] Handling call
0806 20:35:37.428047 (+891712us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54003c5354. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:37.428210 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1979578240}
I20260806 20:35:37.265142 16647 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.438050 14521 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.400036 16820 raft_consensus.cc:3032] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:37.407279 16714 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:37.407933 16705 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:37.408146 16817 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.438258 16524 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:37.461310 16633 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.462684 14369 catalog_manager.cc:5495] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:37.442327 16687 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:37.444223 16820 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:37.479513 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 137343us
I20260806 20:35:37.481112 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260806 20:35:37.485096 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 168840us
W20260806 20:35:37.487175 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 148727us
I20260806 20:35:37.480387 14521 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.483009 16823 consensus_queue.cc:1046] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:37.451406 15153 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.446482 16654 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:37.479427 14596 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.448110 16714 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.489193 14369 catalog_manager.cc:5495] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 3, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 3 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:37.491214 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:35:37.456596 16664 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:37.492470 16647 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:37.456485 16817 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:37.493983 15151 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:37.501005 14441 leader_election.cc:334] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.982s (SENT)
I20260806 20:35:37.501711 14521 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.485314 16687 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:37.478422 16705 raft_consensus.cc:686] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:37.477209 16633 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:37.523034 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 71737us
I20260806 20:35:37.520915 14369 catalog_manager.cc:5495] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 reported cstate change: term changed from 1 to 3, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 3 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 20:35:37.524708 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 141562us
I20260806 20:35:37.525189 14441 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:37.517124 14595 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:37.520967 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 64615.4us
I20260806 20:35:37.515435 16714 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:37.530537 16664 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:37.530972 16705 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.509262 16654 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:37.535815 15153 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.516016 16524 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:37.540557 16624 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.544941 14596 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.545050 16796 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:37.531016 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:35:37.546571 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 45609.5us
I20260806 20:35:37.539947 16696 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.528234 16760 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.532984 16687 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:37.528703 16817 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.536494 16633 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.570122 15153 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.557206 16654 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:37.573685 14441 leader_election.cc:334] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.778s (SENT)
I20260806 20:35:37.580963 14441 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:37.574746 14444 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:37.554004 16524 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:37.557018 16624 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:37.563292 16796 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.567843 16647 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 20:35:37.673662 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 50609.4us
W20260806 20:35:37.620409 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47810.2us
W20260806 20:35:37.620365 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47405.9us
I20260806 20:35:37.628480 16830 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.673306 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1501) took 2901 ms. Trace:
I20260806 20:35:37.569010 16753 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:37.568349 16760 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.584834 16524 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:37.812975 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 295972us
I20260806 20:35:37.813637 16664 consensus_queue.cc:1035] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:37.814170 14595 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:37.815110 16746 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:37.815501 16746 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:37.815872 16746 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:37.816996 16828 log.cc:924] Time spent T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.172s user 0.003s sys 0.000s
I20260806 20:35:37.819352 16664 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:37.821619 16664 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:35:37.822871 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1724) took 5307 ms. Trace:
I20260806 20:35:37.715703 16714 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:37.825196 14595 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.823400 14629 rpcz_store.cc:276] 0806 20:35:32.515972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.033752 (+4517780us) service_pool.cc:227] Handling call
0806 20:35:37.316478 (+282726us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:37.316657 (+ 179us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:37.317102 (+ 445us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:37.317128 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:37.499543 (+182415us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b2c000b26a8. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:37.499976 (+ 433us) log.cc:842] Serialized 26 byte log entry
0806 20:35:37.569158 (+ 69182us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:37.569282 (+ 124us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.569464 (+ 182us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:37.822666 (+253202us) raft_consensus.cc:1671] finished
0806 20:35:37.822687 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.822813 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4131,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":396396672,"thread_start_us":68418,"threads_started":1,"wal-append.queue_time_us":320592}
I20260806 20:35:37.825295 16714 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:37.573040 16683 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:35:37.627367 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 52705.6us
I20260806 20:35:37.828059 16525 consensus_queue.cc:1035] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:37.582630 16820 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:37.826323 16830 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:37.608443 16460 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:37.700217 16624 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.849524 16714 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.849985 16714 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:37.850442 16714 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.620076 16834 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:37.695528 16796 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:37.857211 16830 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.826718 16760 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.573778 16623 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.572655 16815 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:37.868628 16815 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:37.869021 16815 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:37.570981 16823 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:37.699918 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 206025us
I20260806 20:35:37.745371 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1658) took 4678 ms. Trace:
I20260806 20:35:37.708840 16696 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:37.705557 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 48774.9us
I20260806 20:35:37.715531 16687 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:37.749069 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 247448us
I20260806 20:35:37.671389 16657 raft_consensus.cc:3032] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 20:35:37.745692 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 172041us
I20260806 20:35:37.847875 16460 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.824014 16746 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:37.828724 16753 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:37.864490 16714 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:37.867327 16826 log.cc:924] Time spent T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.227s user 0.000s sys 0.001s
I20260806 20:35:37.822625 14478 rpcz_store.cc:276] 0806 20:35:34.755946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.762357 (+2006411us) service_pool.cc:227] Handling call
0806 20:35:36.817168 (+ 54811us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:37.196812 (+379644us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:37.196856 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:37.243186 (+ 46330us) log.cc:842] Serialized 26 byte log entry
0806 20:35:37.480112 (+236926us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:37.480157 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.480227 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:37.657353 (+177126us) raft_consensus.cc:1671] finished
0806 20:35:37.657381 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.657562 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12327,"raft.queue_time_us":30094,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":12785,"thread_start_us":487064,"threads_started":2,"wal-append.queue_time_us":304774}
I20260806 20:35:37.871078 14444 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.870469 14471 rpcz_store.cc:276] 0806 20:35:33.066482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.775934 (+3709452us) service_pool.cc:227] Handling call
0806 20:35:36.861430 (+ 85496us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.229731 (+368301us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:37.450918 (+221187us) log.cc:842] Serialized 14 byte log entry
0806 20:35:37.724620 (+273702us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:37.724690 (+ 70us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:37.745076 (+ 20386us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.745171 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.745308 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44373,"prepare.queue_time_us":285534,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":70044,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":64800,"thread_start_us":712210,"threads_started":4}
I20260806 20:35:37.838217 14596 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.883121 15151 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:37.627720 16286 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.234s
I20260806 20:35:37.715886 16817 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.924494 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1742) took 1265 ms. Trace:
I20260806 20:35:37.925172 14627 rpcz_store.cc:276] 0806 20:35:36.658900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.135318 (+476418us) service_pool.cc:227] Handling call
0806 20:35:37.296936 (+161618us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:37.341421 (+ 44485us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:37.567252 (+225831us) log.cc:842] Serialized 14 byte log entry
0806 20:35:37.924052 (+356800us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:37.924122 (+ 70us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:37.924182 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.924257 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:37.924399 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4483,"prepare.queue_time_us":113192,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":1434,"raft.queue_time_us":11589,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":2432,"thread_start_us":498586,"threads_started":3}
I20260806 20:35:37.928185 14521 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.931669 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:37.887454 16657 raft_consensus.cc:729] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:37.840591 16647 raft_consensus.cc:686] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:37.839135 16683 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:37.904589 16761 consensus_meta.cc:287] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.593s user 0.010s sys 0.002s
I20260806 20:35:37.686822 16811 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.184s
W20260806 20:35:37.935197 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 94123.6us
I20260806 20:35:37.671128 16822 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.271s
I20260806 20:35:37.993515 16683 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:37.915303 16696 consensus_queue.cc:1035] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260806 20:35:37.996765 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:37.969149 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 67714.6us
I20260806 20:35:37.992975 16844 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:38.003418 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.001601 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.010154 14629 raft_consensus.cc:2379] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:37.900396 16460 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:38.002031 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.011658 14554 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:37.885980 14441 leader_election.cc:334] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 2.146s (SENT)
W20260806 20:35:38.012305 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 126365us
I20260806 20:35:37.911458 16753 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:37.895339 16820 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:37.931653 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 88556.3us
I20260806 20:35:38.009456 14521 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:38.014315 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 131288us
I20260806 20:35:38.021191 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
W20260806 20:35:38.017387 14441 leader_election.cc:334] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 2.219s (SENT)
W20260806 20:35:37.953841 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 124799us
I20260806 20:35:38.026000 14552 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
I20260806 20:35:38.022329 14441 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:38.026197 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:37.956617 16841 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:35:38.044509 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 148689us
I20260806 20:35:37.997653 16687 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:37.993139 16623 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:37.934939 16633 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:38.013129 16761 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:37.994485 16647 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:37.999068 16524 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:38.054215 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.121457 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.109469 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.098811 14595 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:38.121079 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:38.021715 16545 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
W20260806 20:35:38.161854 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 148914us
I20260806 20:35:38.161985 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.025421 16844 raft_consensus.cc:686] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:38.183257 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 96972.1us
W20260806 20:35:38.172513 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 139605us
I20260806 20:35:38.025770 16705 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:38.196022 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:38.193864 14441 leader_election.cc:334] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 2.439s (SENT)
I20260806 20:35:38.127045 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.204396 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.033836 16841 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:38.098902 16687 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:38.206362 14555 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:38.207237 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.205768 16705 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:38.209131 14549 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:38.210615 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:38.211042 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.033581 16849 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:38.217109 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.227211 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.121256 16848 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:35:38.160979 16696 consensus_queue.cc:1035] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260806 20:35:38.083657 16657 consensus_queue.cc:260] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:38.420356 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 216652us
I20260806 20:35:38.789822 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:38.366400 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:38.197440 16834 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:38.232565 14471 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:38.231738 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:38.231997 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 210894us
I20260806 20:35:38.392403 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:38.616310 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:38.485720 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:38.808756 14554 raft_consensus.cc:2379] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:38.419837 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:38.333642 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:38.331616 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:38.811749 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.304646 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.812851 14476 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:38.244290 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:38.209004 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:38.168335 16760 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:38.231642 16844 consensus_queue.cc:237] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:38.777361 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:38.609738 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 265083us
I20260806 20:35:39.192195 14553 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:38.418442 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:38.810284 14631 raft_consensus.cc:2362] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6. Re-sending same reply.
I20260806 20:35:39.192562 14629 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:39.046121 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.198558 15187 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:39.201720 14547 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:39.201972 15153 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.202171 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1478) took 1006 ms. Trace:
W20260806 20:35:39.180473 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 330907us
I20260806 20:35:38.391965 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.202770 14547 rpcz_store.cc:276] 0806 20:35:38.195820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.215094 (+ 19274us) service_pool.cc:227] Handling call
0806 20:35:39.202120 (+987026us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.203732 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1598) took 1039 ms. Trace:
I20260806 20:35:39.204090 14554 rpcz_store.cc:276] 0806 20:35:38.185734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.203209 (+ 17475us) service_pool.cc:227] Handling call
0806 20:35:39.203676 (+1000467us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:39.023463 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 145353us
I20260806 20:35:38.787938 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.788398 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.206430 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.005188 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:38.902432 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:38.201714 16623 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:39.209689 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1476) took 1015 ms. Trace:
I20260806 20:35:39.208396 14633 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:39.210163 14553 rpcz_store.cc:276] 0806 20:35:38.194681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.194793 (+ 112us) service_pool.cc:227] Handling call
0806 20:35:39.209633 (+1014840us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.210390 14444 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:39.211687 14627 raft_consensus.cc:2362] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5. Re-sending same reply.
W20260806 20:35:38.808166 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 597615us
I20260806 20:35:39.212342 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 294) took 1254 ms. Trace:
W20260806 20:35:38.608736 16816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.016s sys 0.055s
I20260806 20:35:38.882165 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.213130 14474 raft_consensus.cc:2354] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5.
I20260806 20:35:39.205525 14521 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:39.049738 16460 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.024s user 0.009s sys 0.001s
W20260806 20:35:39.096899 16753 consensus_meta.cc:287] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.065s user 0.002s sys 0.010s
W20260806 20:35:39.277670 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 75785.2us
W20260806 20:35:39.108677 14441 leader_election.cc:334] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 2.745s (SENT)
I20260806 20:35:39.245846 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:39.266023 14627 rpcz_store.cc:276] 0806 20:35:37.958108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.244009 (+285901us) service_pool.cc:227] Handling call
0806 20:35:39.212272 (+968263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.272352 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:38.392055 15181 consensus_meta.cc:287] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.541s user 0.008s sys 0.003s
W20260806 20:35:39.275379 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 65099.1us
I20260806 20:35:39.324101 14547 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:38.640765 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.272946 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:38.419381 16846 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.208s user 0.000s sys 0.000s
W20260806 20:35:39.053481 16796 consensus_meta.cc:287] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.200s user 0.010s sys 0.000s
W20260806 20:35:39.054898 16624 consensus_meta.cc:287] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.206s user 0.010s sys 0.000s
I20260806 20:35:39.191246 16657 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:39.043002 16654 consensus_meta.cc:287] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.429s user 0.012s sys 0.000s
W20260806 20:35:38.238898 16836 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.222s user 0.000s sys 0.000s
I20260806 20:35:38.814062 16705 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.200482 14552 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
W20260806 20:35:38.811685 16822 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.008s sys 0.009s
I20260806 20:35:39.220407 14477 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:39.273859 14518 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 2.142s (SENT) [suppressed 38 similar messages]
I20260806 20:35:39.024669 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:38.398030 16830 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:39.428864 16624 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:39.441632 16830 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:38.237599 16843 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.221s user 0.000s sys 0.000s
W20260806 20:35:39.369992 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 164556us
I20260806 20:35:39.443038 16624 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:38.258203 16524 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:38.813216 16849 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:39.443857 16624 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:39.444017 16849 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:39.006902 16818 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.008s sys 0.022s
I20260806 20:35:39.364818 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.424640 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.275640 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1479) took 1070 ms. Trace:
I20260806 20:35:38.660029 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.320585 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.364317 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 300) took 1055 ms. Trace:
I20260806 20:35:39.494592 14632 rpcz_store.cc:276] 0806 20:35:38.277075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.277244 (+ 169us) service_pool.cc:227] Handling call
0806 20:35:38.277715 (+ 471us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:38.330640 (+ 52925us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:38.330671 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:38.331178 (+ 507us) log.cc:842] Serialized 26 byte log entry
0806 20:35:39.174647 (+843469us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:39.174693 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:39.174780 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:39.332695 (+157915us) raft_consensus.cc:1671] finished
0806 20:35:39.332715 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:39.332858 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":66197,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":112,"thread_start_us":398131,"threads_started":1,"wal-append.queue_time_us":984206}
W20260806 20:35:39.323402 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 214779us
I20260806 20:35:39.364271 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.278616 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.365517 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.333655 15153 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.366215 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:39.523914 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 76014.3us
I20260806 20:35:39.532059 14547 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 5
I20260806 20:35:39.512302 14555 rpcz_store.cc:276] 0806 20:35:38.205603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.216830 (+ 11227us) service_pool.cc:227] Handling call
0806 20:35:39.275570 (+1058740us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.334226 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:38.922008 15180 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:38.344941 16714 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:38.211917 16841 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:39.329053 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.541751 14471 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:39.188325 16851 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:39.365823 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.333870 14548 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
W20260806 20:35:39.072782 16820 consensus_meta.cc:287] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.040s user 0.011s sys 0.000s
W20260806 20:35:38.238349 16845 log.cc:924] Time spent T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260806 20:35:39.364782 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.429694 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:38.260795 16746 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:39.431702 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1599) took 1237 ms. Trace:
I20260806 20:35:39.547204 14552 rpcz_store.cc:276] 0806 20:35:38.194206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.194366 (+ 160us) service_pool.cc:227] Handling call
0806 20:35:39.431597 (+1237231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.254120 16834 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:39.321069 16816 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:39.036165 16848 raft_consensus.cc:686] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:39.433985 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 230849us
I20260806 20:35:39.434864 14442 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:39.321460 16286 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:39.028966 16852 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:39.536584 15183 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:39.368172 16687 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.149s user 0.009s sys 0.000s
I20260806 20:35:39.005343 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.273226 16647 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:39.431567 16696 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:39.438771 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 525) took 1399 ms. Trace:
I20260806 20:35:39.439033 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1668) took 1104 ms. Trace:
I20260806 20:35:39.570942 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.570886 14477 rpcz_store.cc:276] 0806 20:35:38.334201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.367210 (+ 33009us) service_pool.cc:227] Handling call
0806 20:35:39.438972 (+1071762us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.571674 14552 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:39.430832 16654 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:39.439448 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.364639 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:38.475596 16817 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:39.576297 16851 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:39.445003 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 120490us
I20260806 20:35:39.576786 16851 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:39.577208 14444 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.578127 14444 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.578996 14444 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.579403 16852 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:38.227571 16545 raft_consensus.cc:686] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:39.580597 14550 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:39.580905 16545 consensus_queue.cc:237] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:39.581667 16852 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:39.583098 16848 consensus_queue.cc:237] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:39.583585 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:39.583911 16853 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:39.584331 16853 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:39.584705 16853 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.585309 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.005188 16761 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:39.449189 16624 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:39.332885 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:38.922827 16633 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:39.454008 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1399) took 3661 ms. Trace:
I20260806 20:35:39.425166 16460 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:38.246719 16525 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 1.184s
I20260806 20:35:39.368477 14469 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:39.428323 16796 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:39.484974 16815 consensus_meta.cc:287] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.615s user 0.009s sys 0.005s
W20260806 20:35:39.487551 16823 consensus_meta.cc:287] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.604s user 0.011s sys 0.000s
I20260806 20:35:38.250645 16683 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:39.433374 16822 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.430s user 0.025s sys 0.029s
I20260806 20:35:39.511337 16856 consensus_queue.cc:1035] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:39.503249 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:39.504138 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:39.510591 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 65590.7us
I20260806 20:35:39.531299 14554 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:39.503057 14553 raft_consensus.cc:2379] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:39.347626 15181 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:39.503768 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.365139 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.533107 16786 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:39.533495 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 199940us
I20260806 20:35:39.533766 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:39.538122 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 325062us
W20260806 20:35:39.525493 16818 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.548s user 0.011s sys 0.022s
W20260806 20:35:39.538991 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 170307us
I20260806 20:35:39.539413 14596 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.543555 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.333599 16753 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:39.550894 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.557484 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:39.565239 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:39.566579 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:39.567736 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 296) took 1307 ms. Trace:
I20260806 20:35:39.568545 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:39.569306 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 137365us
I20260806 20:35:39.569891 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:39.562882 16834 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.548504 16854 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:39.579800 14444 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:39.213817 16623 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:39.562491 16816 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:39.599458 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.610044 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:39.570659 15179 rpcz_store.cc:276] 0806 20:35:38.034127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.076936 (+ 42809us) service_pool.cc:227] Handling call
0806 20:35:39.432045 (+1355109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.773586 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.773720 16714 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:39.773874 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.774268 16714 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:39.774669 16714 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.774907 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.775127 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.775051 14625 rpcz_store.cc:276] 0806 20:35:38.260509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.330738 (+ 70229us) service_pool.cc:227] Handling call
0806 20:35:38.810689 (+479951us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:38.883266 (+ 72577us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:39.273922 (+390656us) log.cc:842] Serialized 14 byte log entry
0806 20:35:39.567355 (+293433us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:39.567405 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:39.567468 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:39.567559 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:39.567677 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":554665,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":112144,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":4780,"spinlock_wait_cycles":241536,"thread_start_us":545860,"threads_started":3,"wal-append.queue_time_us":164251}
I20260806 20:35:39.762640 16855 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:39.776633 16623 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.776961 16855 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:39.777387 16855 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:39.777848 14471 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:39.779543 16823 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:39.780118 14478 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:39.780767 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1670) took 1381 ms. Trace:
I20260806 20:35:39.781279 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1666) took 1551 ms. Trace:
I20260806 20:35:39.781551 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.782059 14626 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 20:35:39.783176 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:39.783893 14548 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.781818 14478 rpcz_store.cc:276] 0806 20:35:38.229538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.231464 (+ 1926us) service_pool.cc:227] Handling call
0806 20:35:39.781124 (+1549660us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54003c6254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:39.781225 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332907392}
I20260806 20:35:39.573920 15178 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:39.780789 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.783380 14474 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:39.607985 14628 rpcz_store.cc:276] 0806 20:35:35.792309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:37.042284 (+1249975us) service_pool.cc:227] Handling call
0806 20:35:37.303614 (+261330us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593b7c 00007fb9b059373b 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:37.303913 (+ 299us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:37.304182 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:37.304210 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:37.508829 (+204619us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b2c000b26a8. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:37.521606 (+ 12777us) log.cc:842] Serialized 26 byte log entry
0806 20:35:37.907786 (+386180us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:37.907822 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:37.907914 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:39.453708 (+1545794us) raft_consensus.cc:1671] finished
0806 20:35:39.453734 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:39.453914 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84889856,"thread_start_us":337369,"threads_started":1,"wal-append.queue_time_us":493630}
I20260806 20:35:39.699821 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.774763 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:39.786295 14470 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:39.773775 14552 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:39.657639 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 125702us
I20260806 20:35:39.658352 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.776300 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:39.572944 16687 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:39.789772 15184 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
W20260806 20:35:39.787009 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 247675us
I20260806 20:35:39.657223 14547 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:39.790189 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.573328 16820 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:39.792872 14553 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:39.777207 14442 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:39.825373 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:39.799765 15153 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.799579 14595 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:39.827916 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:39.800437 15150 leader_election.cc:334] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.596s (SENT)
I20260806 20:35:39.828866 14475 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:35:39.828971 14441 leader_election.cc:334] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.957s (SENT)
I20260806 20:35:39.606069 16624 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.834717 14549 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:39.829384 14518 connection.cc:232] RPC call timeout handler was delayed by -4.5217373790219426s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33940000000000003s for a response
I20260806 20:35:39.598822 16760 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:39.780000 16815 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:39.856730 14470 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:39.830039 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:39.858103 14518 connection.cc:232] RPC call timeout handler was delayed by -3.9137404820066877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2891s for a response
W20260806 20:35:39.832620 15151 connection.cc:232] RPC call timeout handler was delayed by -4.0370616019936278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20260806 20:35:39.830390 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:39.857522 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 57124.4us
I20260806 20:35:39.761207 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.859524 16654 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:39.860252 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:39.861541 15181 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 450) took 2967 ms (client timeout 1853 ms). Trace:
I20260806 20:35:39.862023 16705 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:39.864677 14473 raft_consensus.cc:2396] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:39.865937 16849 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:39.866879 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.855809 16761 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:35:39.829530 15178 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 527) took 1635 ms (client timeout 1982 ms). Trace:
I20260806 20:35:39.787585 16786 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:39.788262 16816 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:39.781142 14469 rpcz_store.cc:276] 0806 20:35:38.399350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.475316 (+ 75966us) service_pool.cc:227] Handling call
0806 20:35:39.780701 (+1305385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.830039 16818 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 20:35:39.876861 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 99780.7us
I20260806 20:35:39.877599 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:39.879743 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:39.879984 14633 consensus_meta.cc:287] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.589s user 0.006s sys 0.001s
I20260806 20:35:39.881749 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.606594 16817 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:39.690024 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:39.830678 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 250933us
I20260806 20:35:39.798977 16854 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:39.920419 15151 connection.cc:232] RPC call timeout handler was delayed by -3.9713616019871552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
W20260806 20:35:39.920553 14518 connection.cc:232] RPC call timeout handler was delayed by -3.5149792230222374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20260806 20:35:39.922006 16815 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:39.831441 15186 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:39.922210 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:39.831758 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:39.826151 16820 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:39.830555 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:39.776381 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:39.923668 16286 raft_consensus.cc:686] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:39.857103 16823 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:39.924382 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 270) took 1072 ms. Trace:
I20260806 20:35:39.859099 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1608) took 1011 ms. Trace:
I20260806 20:35:39.924961 14548 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:39.926913 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 95664us
I20260806 20:35:39.573256 16647 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:39.928229 14596 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:39.930034 14475 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:39.925324 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 141259us
W20260806 20:35:39.952569 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 153085us
I20260806 20:35:39.617200 16696 raft_consensus.cc:686] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:39.657804 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.958341 15187 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:39.782171 16714 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.223991 16796 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.139762 16624 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.386374 14442 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:40.387254 16888 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:40.387527 16624 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:40.365774 14518 connection.cc:232] RPC call timeout handler was delayed by -3.4654792230285238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
W20260806 20:35:39.953739 15181 rpcz_store.cc:269] 0806 20:35:36.894265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:36.899505 (+ 5240us) service_pool.cc:227] Handling call
0806 20:35:39.861304 (+2961799us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b540045db54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:39.861416 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2168773888}
W20260806 20:35:40.392711 14518 connection.cc:232] RPC call timeout handler was delayed by -3.423179223027546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1685s for a response
W20260806 20:35:40.393380 14518 connection.cc:232] RPC call timeout handler was delayed by -3.29987922302098s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
W20260806 20:35:40.393824 14518 connection.cc:232] RPC call timeout handler was delayed by -3.2170792230172083s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260806 20:35:40.394193 14518 connection.cc:232] RPC call timeout handler was delayed by -3.0388792230223771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
W20260806 20:35:40.394572 14518 connection.cc:232] RPC call timeout handler was delayed by -2.9353792230249383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
W20260806 20:35:40.394919 14518 connection.cc:232] RPC call timeout handler was delayed by -2.6842792230308987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2506s for a response
I20260806 20:35:39.931102 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.366228 16820 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:40.052907 16753 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.252012 16623 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.396873 14630 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:35:40.398015 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 299) took 2133 ms. Trace:
I20260806 20:35:40.398316 14630 rpcz_store.cc:276] 0806 20:35:38.264486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.331284 (+ 66798us) service_pool.cc:227] Handling call
0806 20:35:38.808797 (+477513us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.397221 (+1588424us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.397786 (+ 565us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54002404d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:40.397962 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3651735296}
I20260806 20:35:40.393337 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.402149 16623 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.254245 16849 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.403851 16820 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.406044 16820 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:40.054358 14369 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56582 (request call id 139) took 14586 ms (client timeout 15000 ms). Trace:
I20260806 20:35:40.369067 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 558) took 1091 ms. Trace:
I20260806 20:35:40.407370 15179 rpcz_store.cc:276] 0806 20:35:39.277688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.776116 (+498428us) service_pool.cc:227] Handling call
0806 20:35:40.368992 (+592876us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.408434 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.409197 15179 raft_consensus.cc:2354] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6.
I20260806 20:35:40.367847 16818 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:40.100587 14624 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:40.409590 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 540) took 1382 ms. Trace:
I20260806 20:35:40.410010 15179 rpcz_store.cc:276] 0806 20:35:39.027501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.408153 (+1380652us) service_pool.cc:227] Handling call
0806 20:35:40.409536 (+ 1383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.410571 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 302) took 1591 ms. Trace:
I20260806 20:35:40.411021 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.410969 14624 rpcz_store.cc:276] 0806 20:35:38.819054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.840924 (+ 21870us) service_pool.cc:227] Handling call
0806 20:35:39.042149 (+201225us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.409705 (+1367556us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.410322 (+ 617us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b5400313654. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:40.410508 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2255982976}
I20260806 20:35:40.369730 14633 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:40.411990 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 557) took 1140 ms. Trace:
I20260806 20:35:40.025703 14550 raft_consensus.cc:2379] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:40.370149 16714 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.252601 16705 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.370546 14555 raft_consensus.cc:2379] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:40.414125 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1613) took 1139 ms. Trace:
I20260806 20:35:40.414266 16753 consensus_queue.cc:1046] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:40.414500 14555 rpcz_store.cc:276] 0806 20:35:39.274872 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.921909 (+647037us) service_pool.cc:227] Handling call
0806 20:35:40.414054 (+492145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.369191 14554 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.415315 16815 raft_consensus.cc:3032] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:40.415676 16815 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:40.415998 14555 raft_consensus.cc:1226] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:40.099659 14471 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:40.416865 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 281) took 1085 ms. Trace:
I20260806 20:35:40.416561 16815 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.417210 14471 rpcz_store.cc:276] 0806 20:35:39.331411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.542527 (+211116us) service_pool.cc:227] Handling call
0806 20:35:40.416806 (+874279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.415122 15179 rpcz_store.cc:276] 0806 20:35:39.272158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.410731 (+1138573us) service_pool.cc:227] Handling call
0806 20:35:40.411927 (+ 1196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.418452 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.372176 16870 consensus_queue.cc:1046] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:35:40.371412 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 537) took 1344 ms. Trace:
I20260806 20:35:40.419253 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 378) took 1092 ms. Trace:
I20260806 20:35:40.419546 16870 raft_consensus.cc:3032] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:40.247637 16855 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.419776 15179 rpcz_store.cc:276] 0806 20:35:39.327228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.418203 (+1090975us) service_pool.cc:227] Handling call
0806 20:35:40.419204 (+ 1001us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.420047 16870 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:40.420912 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.421049 16870 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:39.977784 16874 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:40.422228 16874 raft_consensus.cc:686] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:40.422432 16870 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:40.423739 16815 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:39.973569 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1610) took 1125 ms. Trace:
W20260806 20:35:40.373590 15151 connection.cc:232] RPC call timeout handler was delayed by -3.5447616019810084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
W20260806 20:35:40.424435 15151 connection.cc:232] RPC call timeout handler was delayed by -3.4853616019827314s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2136s for a response
I20260806 20:35:40.424340 14549 rpcz_store.cc:276] 0806 20:35:38.848021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.789914 (+941893us) service_pool.cc:227] Handling call
0806 20:35:39.973323 (+183409us) spinlock_profiling.cc:243] Waited 8.06 ms on lock 0x7b540013abd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:39.973495 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18544512}
W20260806 20:35:40.424844 15151 connection.cc:232] RPC call timeout handler was delayed by -3.4655616020027082s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2158s for a response
W20260806 20:35:40.425151 15151 connection.cc:232] RPC call timeout handler was delayed by -2.7509616019960959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2952s for a response
I20260806 20:35:40.372850 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 551) took 1301 ms. Trace:
I20260806 20:35:40.425793 15182 rpcz_store.cc:276] 0806 20:35:39.071547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.605920 (+534373us) service_pool.cc:227] Handling call
0806 20:35:40.372790 (+766870us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.426712 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.427466 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 377) took 1100 ms. Trace:
I20260806 20:35:40.427775 15182 rpcz_store.cc:276] 0806 20:35:39.326950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.426452 (+1099502us) service_pool.cc:227] Handling call
0806 20:35:40.427389 (+ 937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:39.978168 15180 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:40.374318 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1690) took 1266 ms. Trace:
I20260806 20:35:40.428666 14477 rpcz_store.cc:276] 0806 20:35:39.107783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.786431 (+678648us) service_pool.cc:227] Handling call
0806 20:35:40.374263 (+587832us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.429786 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1484) took 1096 ms. Trace:
I20260806 20:35:40.430050 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 376) took 1103 ms. Trace:
I20260806 20:35:40.430203 14555 rpcz_store.cc:276] 0806 20:35:39.332965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.415508 (+1082543us) service_pool.cc:227] Handling call
0806 20:35:40.415828 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.416401 (+ 573us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.429732 (+ 13331us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.430754 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.419204 15183 rpcz_store.cc:276] 0806 20:35:39.026535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.594917 (+568382us) service_pool.cc:227] Handling call
0806 20:35:40.371348 (+776431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.432335 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:39.925863 16873 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.449s
I20260806 20:35:40.433071 16822 consensus_queue.cc:1046] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:35:40.433482 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:40.433837 15151 leader_election.cc:334] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.136s (SENT)
W20260806 20:35:40.375381 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 474607us
I20260806 20:35:40.434351 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 546) took 1365 ms. Trace:
I20260806 20:35:40.434710 15183 rpcz_store.cc:276] 0806 20:35:39.069424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.433250 (+1363826us) service_pool.cc:227] Handling call
0806 20:35:40.434300 (+ 1050us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.435019 14596 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.436084 16714 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:40.436591 16714 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.373440 14478 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
W20260806 20:35:40.437156 14441 leader_election.cc:334] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.975s (SENT)
I20260806 20:35:40.436107 14596 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.437489 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 277) took 1263 ms. Trace:
I20260806 20:35:40.413249 16818 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.437847 14478 rpcz_store.cc:276] 0806 20:35:39.173838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.784772 (+610934us) service_pool.cc:227] Handling call
0806 20:35:40.437431 (+652659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.438338 16714 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.439268 14478 raft_consensus.cc:1261] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:40.439790 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1681) took 1342 ms. Trace:
I20260806 20:35:40.375470 16545 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:40.440646 16545 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.440691 16823 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:40.374361 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.368175 16286 consensus_queue.cc:237] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:40.365272 15186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 526) took 2263 ms (client timeout 2146 ms). Trace:
W20260806 20:35:40.446254 15186 rpcz_store.cc:269] 0806 20:35:38.102083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.120728 (+ 18645us) service_pool.cc:227] Handling call
0806 20:35:40.365071 (+2244343us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54005cd354. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.365201 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3830910336}
W20260806 20:35:40.380659 16834 consensus_meta.cc:287] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.555s user 0.003s sys 0.007s
I20260806 20:35:40.380573 14595 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:40.380913 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.450007 14625 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.381695 16696 consensus_queue.cc:237] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.382272 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1677) took 1504 ms. Trace:
I20260806 20:35:40.382444 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.363086 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1620) took 1043 ms. Trace:
I20260806 20:35:40.383845 16854 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.384160 16647 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.047405 16872 raft_consensus.cc:3032] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:40.373955 16817 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:40.341737 16683 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.387712 16888 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.388839 14476 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:40.327029 14551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1609) took 1479 ms (client timeout 1829 ms). Trace:
W20260806 20:35:40.492259 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 111850us
W20260806 20:35:40.492808 14551 rpcz_store.cc:269] 0806 20:35:38.847725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.016882 (+169157us) service_pool.cc:227] Handling call
0806 20:35:40.326805 (+1309923us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000b31d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.326951 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2586511744}
W20260806 20:35:39.960857 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 178002us
I20260806 20:35:40.487601 16823 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.494771 14521 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.496083 16823 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.389827 14547 rpcz_store.cc:276] 0806 20:35:38.847437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.827679 (+980242us) service_pool.cc:227] Handling call
0806 20:35:39.859027 (+ 31348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.456280 14632 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.497360 14547 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:40.497941 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1487) took 1132 ms. Trace:
I20260806 20:35:40.498224 14547 rpcz_store.cc:276] 0806 20:35:39.365962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.496779 (+1130817us) service_pool.cc:227] Handling call
0806 20:35:40.497154 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.497707 (+ 553us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.497887 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:40.390307 15178 rpcz_store.cc:269] 0806 20:35:38.189292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.206054 (+ 16762us) service_pool.cc:227] Handling call
0806 20:35:39.824710 (+1618656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.390949 15184 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:40.391789 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.502183 14631 raft_consensus.cc:2362] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5. Re-sending same reply.
I20260806 20:35:40.395629 14518 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:40.123816 16851 consensus_meta.cc:287] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.547s user 0.011s sys 0.001s
I20260806 20:35:40.402662 15181 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:40.366948 14444 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.430471 15180 rpcz_store.cc:276] 0806 20:35:39.326683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.866583 (+539900us) service_pool.cc:227] Handling call
0806 20:35:40.429986 (+563403us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:40.503964 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 137093us
I20260806 20:35:40.051918 16460 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.406181 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.506592 14627 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:40.406817 15185 raft_consensus.cc:2379] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
W20260806 20:35:40.412395 14633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 298) took 2148 ms (client timeout 1892 ms). Trace:
I20260806 20:35:40.508666 14632 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:40.508710 14444 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:40.509012 14633 rpcz_store.cc:269] 0806 20:35:38.264144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.330655 (+ 66511us) service_pool.cc:227] Handling call
0806 20:35:40.412303 (+2081648us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.369840 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 271) took 1517 ms. Trace:
I20260806 20:35:40.414449 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1612) took 1139 ms. Trace:
W20260806 20:35:40.510440 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 68412us
I20260806 20:35:40.418574 14471 raft_consensus.cc:1261] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:40.510879 14444 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:40.421545 16855 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:40.259485 16853 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.674s user 0.008s sys 0.001s
I20260806 20:35:40.423199 16874 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.343030 14553 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:40.373914 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:40.357287 16841 consensus_meta.cc:287] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.793s user 0.010s sys 0.001s
W20260806 20:35:39.962828 14473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1672) took 1477 ms (client timeout 1728 ms). Trace:
I20260806 20:35:40.499486 16888 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.367656 14470 rpcz_store.cc:276] 0806 20:35:38.852178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.806578 (+954400us) service_pool.cc:227] Handling call
0806 20:35:39.856575 (+ 49997us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:39.924153 (+ 67578us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:39.924329 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.440068 14478 rpcz_store.cc:276] 0806 20:35:39.097288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.438648 (+1341360us) service_pool.cc:227] Handling call
0806 20:35:40.438957 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.439577 (+ 620us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.439747 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.375821 16786 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:40.438076 14441 leader_election.cc:334] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 2.037s (SENT)
I20260806 20:35:40.445438 15187 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:40.388118 14442 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:40.453089 14596 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.436643 16844 raft_consensus.cc:3032] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:40.441143 16545 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.524515 16844 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:40.526419 16848 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:40.526948 16844 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.528630 16683 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:40.421576 15179 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:40.529029 16683 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.529249 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 545) took 1460 ms. Trace:
I20260806 20:35:40.529418 16683 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.529639 15179 rpcz_store.cc:276] 0806 20:35:39.069058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.420652 (+1351594us) service_pool.cc:227] Handling call
0806 20:35:40.529190 (+108538us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.530679 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.531271 15179 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:40.531711 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 535) took 1505 ms. Trace:
I20260806 20:35:40.456229 14552 rpcz_store.cc:276] 0806 20:35:39.320041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.922823 (+602782us) service_pool.cc:227] Handling call
0806 20:35:39.923115 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:39.923352 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:40.180810 (+257458us) log.cc:842] Serialized 14 byte log entry
0806 20:35:40.362549 (+181739us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.362622 (+ 73us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:40.362703 (+ 81us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.362794 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.362977 (+ 183us) inbound_call.cc:173] Queueing success response
0806 20:35:40.381384 (+ 18407us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b1800368620. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb764 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
Metrics: {"prepare.queue_time_us":402826,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":105,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"spinlock_wait_cycles":32204416,"thread_start_us":395780,"threads_started":2,"wal-append.queue_time_us":139163}
I20260806 20:35:40.488715 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.455842 14469 rpcz_store.cc:276] 0806 20:35:38.878342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.365969 (+1487627us) service_pool.cc:227] Handling call
0806 20:35:40.366319 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.377888 (+ 11569us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:40.377926 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.377967 (+ 41us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:40.378037 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.382021 (+ 3984us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5400332cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:40.382061 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.382195 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6989,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":25849088}
I20260806 20:35:40.533201 15186 raft_consensus.cc:2396] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:40.533628 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 380) took 1205 ms. Trace:
I20260806 20:35:40.533834 14469 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:40.532011 15179 rpcz_store.cc:276] 0806 20:35:39.025859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.530381 (+1504522us) service_pool.cc:227] Handling call
0806 20:35:40.531664 (+ 1283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.533941 15186 rpcz_store.cc:276] 0806 20:35:39.327781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.447242 (+1119461us) service_pool.cc:227] Handling call
0806 20:35:40.533577 (+ 86335us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.518035 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.534687 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1688) took 1427 ms. Trace:
W20260806 20:35:40.412971 14369 rpcz_store.cc:269] 0806 20:35:25.442717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:25.448818 (+ 6101us) service_pool.cc:227] Handling call
0806 20:35:40.023714 (+14574896us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:38.166860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:38.257080 (+ 90220us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:38.257132 (+ 52us) write_op.cc:435] Acquired schema lock
0806 20:35:38.257146 (+ 14us) tablet.cc:660] Decoding operations
0806 20:35:38.258355 (+ 1209us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:38.258467 (+ 112us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:38.258492 (+ 25us) tablet.cc:683] Acquiring locks for 12 operations
0806 20:35:38.420044 (+161552us) tablet.cc:699] Row locks acquired
0806 20:35:38.420087 (+ 43us) write_op.cc:260] PREPARE: finished
0806 20:35:38.615150 (+195063us) write_op.cc:270] Start()
0806 20:35:38.615364 (+ 214us) write_op.cc:276] Timestamp: P: 1786048538420241 usec, L: 0
0806 20:35:38.615405 (+ 41us) op_driver.cc:331] REPLICATION: starting
0806 20:35:39.022784 (+407379us) log.cc:842] Serialized 4850 byte log entry
0806 20:35:39.550654 (+527870us) op_driver.cc:444] REPLICATION: finished
0806 20:35:39.743327 (+192673us) write_op.cc:301] APPLY: starting
0806 20:35:39.754602 (+ 11275us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:39.862579 (+107977us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:39.862732 (+ 153us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:39.864173 (+ 1441us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:39.864568 (+ 395us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0806 20:35:39.864612 (+ 44us) write_op.cc:312] APPLY: finished
0806 20:35:39.864996 (+ 384us) log.cc:842] Serialized 89 byte log entry
0806 20:35:39.865198 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:39.865338 (+ 140us) write_op.cc:454] Released schema lock
0806 20:35:39.865406 (+ 68us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:39.865744 (+ 338us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:39.865776 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":355897,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":132721,"apply.run_cpu_time_us":4401,"apply.run_wall_time_us":164146,"num_ops":12,"prepare.queue_time_us":348462,"prepare.run_cpu_time_us":29674,"prepare.run_wall_time_us":1072915,"raft.queue_time_us":86510,"raft.run_cpu_time_us":11436,"raft.run_wall_time_us":29201,"replication_time_us":935140,"thread_start_us":371330,"threads_started":4,"wal-append.queue_time_us":400950}]]}
I20260806 20:35:40.534942 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.516870 16796 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:40.507310 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:40.535888 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 383) took 1103 ms. Trace:
I20260806 20:35:40.520632 16841 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.534866 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.509987 14475 rpcz_store.cc:276] 0806 20:35:38.852542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.929564 (+1077022us) service_pool.cc:227] Handling call
0806 20:35:39.929867 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.369593 (+439726us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.369760 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.513279 16820 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:40.514391 16854 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:40.539295 16854 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.539642 16854 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.492619 16817 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.510483 14550 rpcz_store.cc:276] 0806 20:35:39.274613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.587414 (+312801us) service_pool.cc:227] Handling call
0806 20:35:40.414278 (+826864us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b5400356e54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.414398 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463366912}
I20260806 20:35:40.444564 16849 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:40.540647 16849 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:40.540982 14550 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:40.541072 16849 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.541666 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1491) took 1090 ms. Trace:
I20260806 20:35:40.541983 14550 rpcz_store.cc:276] 0806 20:35:39.451521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.540404 (+1088883us) service_pool.cc:227] Handling call
0806 20:35:40.540767 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.541419 (+ 652us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.541617 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.543426 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1492) took 1091 ms. Trace:
I20260806 20:35:40.511297 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1678) took 1497 ms. Trace:
I20260806 20:35:40.513473 15181 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 375) took 1184 ms. Trace:
I20260806 20:35:40.544595 14471 rpcz_store.cc:276] 0806 20:35:39.013777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.417921 (+1404144us) service_pool.cc:227] Handling call
0806 20:35:40.418239 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.511049 (+ 92810us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.511248 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.544809 15181 rpcz_store.cc:276] 0806 20:35:39.326406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.393076 (+1066670us) service_pool.cc:227] Handling call
0806 20:35:40.511121 (+118045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.512497 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:40.513652 14551 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 20:35:40.546516 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1489) took 1097 ms. Trace:
I20260806 20:35:40.546823 14595 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:40.546877 14551 rpcz_store.cc:276] 0806 20:35:39.448967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.501231 (+1052264us) service_pool.cc:227] Handling call
0806 20:35:40.513473 (+ 12242us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.546220 (+ 32747us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.546458 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.547972 16624 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:40.548350 16624 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.548741 16624 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.553879 16844 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.514911 14628 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.516475 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 372) took 1191 ms. Trace:
I20260806 20:35:40.554575 15185 rpcz_store.cc:276] 0806 20:35:39.325453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.930826 (+605373us) service_pool.cc:227] Handling call
0806 20:35:40.514547 (+583721us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:40.555047 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 159499us
I20260806 20:35:40.555500 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.556283 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 373) took 1230 ms. Trace:
I20260806 20:35:40.556620 15185 rpcz_store.cc:276] 0806 20:35:39.325840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.555250 (+1229410us) service_pool.cc:227] Handling call
0806 20:35:40.556231 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.557451 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.558714 15179 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:40.558621 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.534844 15178 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.458294 14554 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:40.514837 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1627) took 1061 ms. Trace:
W20260806 20:35:40.516868 14473 rpcz_store.cc:269] 0806 20:35:38.485217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:38.485353 (+ 136us) service_pool.cc:227] Handling call
0806 20:35:39.962581 (+1477228us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b540017e154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:39.962754 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1506119552}
I20260806 20:35:40.515360 14549 raft_consensus.cc:1226] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 20:35:40.435643 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.519855 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.560192 14553 rpcz_store.cc:276] 0806 20:35:39.453679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.776096 (+322417us) service_pool.cc:227] Handling call
0806 20:35:40.514778 (+738682us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.560191 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1481) took 1238 ms. Trace:
I20260806 20:35:40.517938 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.510177 16823 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:40.523100 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 70126.6us
I20260806 20:35:40.524889 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 544) took 1456 ms. Trace:
I20260806 20:35:40.528422 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.580032 14627 raft_consensus.cc:2379] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:40.536307 15186 rpcz_store.cc:276] 0806 20:35:39.432768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.534713 (+1101945us) service_pool.cc:227] Handling call
0806 20:35:40.535829 (+ 1116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.581691 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.538589 16841 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.554008 15182 raft_consensus.cc:2396] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 20:35:40.488416 16872 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:40.583313 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 386) took 1117 ms. Trace:
I20260806 20:35:40.583187 16872 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.583762 15182 rpcz_store.cc:276] 0806 20:35:39.465765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.428380 (+962615us) service_pool.cc:227] Handling call
0806 20:35:40.583115 (+154735us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b540044e154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.583255 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":281208192}
I20260806 20:35:40.584352 16872 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.585290 14625 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:40.585335 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.586602 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 552) took 1514 ms. Trace:
I20260806 20:35:40.586980 15182 rpcz_store.cc:276] 0806 20:35:39.071862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.584671 (+1512809us) service_pool.cc:227] Handling call
0806 20:35:40.586548 (+ 1877us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.506214 16623 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:40.457635 16834 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:40.559482 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 121427us
I20260806 20:35:40.561479 14554 rpcz_store.cc:276] 0806 20:35:39.322273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.825208 (+502935us) service_pool.cc:227] Handling call
0806 20:35:40.560134 (+734926us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.588186 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:40.588840 14441 leader_election.cc:334] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 3.048s (SENT)
I20260806 20:35:40.589478 16834 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.511935 14444 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:40.590521 14479 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:40.590431 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 534) took 1565 ms. Trace:
I20260806 20:35:40.590870 15182 rpcz_store.cc:276] 0806 20:35:39.025402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.589584 (+1564182us) service_pool.cc:227] Handling call
0806 20:35:40.589875 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.590083 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:40.590116 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.590157 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:40.590203 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.590260 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.590359 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:40.591496 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 273) took 1738 ms (client timeout 2171 ms). Trace:
W20260806 20:35:40.591874 14479 rpcz_store.cc:269] 0806 20:35:38.853245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.556026 (+702781us) service_pool.cc:227] Handling call
0806 20:35:40.591332 (+1035306us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b5400331dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.591447 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1853114112}
I20260806 20:35:40.579180 15187 rpcz_store.cc:276] 0806 20:35:39.068659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.374122 (+1305463us) service_pool.cc:227] Handling call
0806 20:35:40.524843 (+150721us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.511255 15153 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.572551 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.521680 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.577543 15184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 549) took 1506 ms. Trace:
I20260806 20:35:40.535056 14469 rpcz_store.cc:276] 0806 20:35:39.107134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.533339 (+1426205us) service_pool.cc:227] Handling call
0806 20:35:40.533653 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.534457 (+ 804us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.534636 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.585531 15186 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
I20260806 20:35:40.588215 16623 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.543816 14550 rpcz_store.cc:276] 0806 20:35:39.451963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.542676 (+1090713us) service_pool.cc:227] Handling call
0806 20:35:40.542968 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.543134 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:40.543167 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.543204 (+ 37us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:40.543240 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.543287 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.543379 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.568986 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1485) took 1203 ms. Trace:
I20260806 20:35:40.545862 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.632974 15181 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 553) took 1560 ms. Trace:
I20260806 20:35:40.633328 15181 rpcz_store.cc:276] 0806 20:35:39.072189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.545612 (+1473423us) service_pool.cc:227] Handling call
0806 20:35:40.632914 (+ 87302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.559182 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 389) took 1092 ms. Trace:
I20260806 20:35:40.634068 15179 rpcz_store.cc:276] 0806 20:35:39.466883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.534609 (+1067726us) service_pool.cc:227] Handling call
0806 20:35:40.559121 (+ 24512us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.615283 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 387) took 1149 ms. Trace:
I20260806 20:35:40.634881 15180 rpcz_store.cc:276] 0806 20:35:39.466128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.514235 (+1048107us) service_pool.cc:227] Handling call
0806 20:35:40.615202 (+100967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.636281 16816 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.638760 14474 raft_consensus.cc:2379] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:40.616375 15184 rpcz_store.cc:276] 0806 20:35:39.070842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.826116 (+755274us) service_pool.cc:227] Handling call
0806 20:35:40.577372 (+751256us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b54003d0ed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.577482 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53811072}
W20260806 20:35:40.640257 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 129060us
I20260806 20:35:40.640645 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1482) took 1318 ms. Trace:
I20260806 20:35:40.611644 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:40.612418 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 224415us
I20260806 20:35:40.615334 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.640997 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1619) took 1321 ms. Trace:
I20260806 20:35:40.641185 15153 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.641825 14631 raft_consensus.cc:2396] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:35:40.618086 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 382) took 1289 ms. Trace:
W20260806 20:35:40.640192 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 59830.4us
I20260806 20:35:40.641048 14548 rpcz_store.cc:276] 0806 20:35:39.322614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.365390 (+1042776us) service_pool.cc:227] Handling call
0806 20:35:40.365766 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.365984 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:40.366023 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:40.366457 (+ 434us) log.cc:842] Serialized 26 byte log entry
0806 20:35:40.508137 (+141680us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:40.508172 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.508300 (+ 128us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:40.640429 (+132129us) raft_consensus.cc:1671] finished
0806 20:35:40.640450 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.640587 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":125625,"threads_started":1,"wal-append.queue_time_us":269975}
I20260806 20:35:40.617255 14549 rpcz_store.cc:276] 0806 20:35:39.365211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.425291 (+1060080us) service_pool.cc:227] Handling call
0806 20:35:40.515171 (+ 89880us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.560646 (+ 45475us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.568781 (+ 8135us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b54001fbdd0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.568928 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":205901568}
I20260806 20:35:40.626456 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:40.626716 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:40.643694 14553 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 1.
I20260806 20:35:40.629699 14632 raft_consensus.cc:2379] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:40.630221 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:40.641530 14546 rpcz_store.cc:276] 0806 20:35:39.319696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.364845 (+ 45149us) service_pool.cc:227] Handling call
0806 20:35:40.640827 (+1275982us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b540013a1d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.640943 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1904129792}
I20260806 20:35:40.642484 15186 rpcz_store.cc:276] 0806 20:35:39.328367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.581421 (+1253054us) service_pool.cc:227] Handling call
0806 20:35:40.618018 (+ 36597us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.596814 14628 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:35:40.628710 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 116836us
I20260806 20:35:40.647363 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.648197 14444 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:40.653213 16817 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.654331 14478 raft_consensus.cc:1226] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:40.654752 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:40.655123 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:40.643613 14554 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:40.656036 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:40.655460 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:40.657302 14549 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:40.658025 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 111285us
I20260806 20:35:40.658718 14595 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.660959 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.662078 16892 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:35:40.662977 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.663213 16834 consensus_queue.cc:1046] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:35:40.662261 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.662632 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:40.509545 16851 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.654757 15153 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.578832 16823 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.668812 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.680745 14633 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:40.677995 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:40.667060 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:40.683060 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:40.683773 14548 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:40.663867 16816 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.518852 16853 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.592411 15182 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:40.672822 16834 raft_consensus.cc:3032] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:40.674566 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 283) took 1342 ms. Trace:
I20260806 20:35:40.612682 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 385) took 1145 ms. Trace:
I20260806 20:35:40.695375 14625 raft_consensus.cc:2354] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8.
I20260806 20:35:40.612149 16841 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:40.678038 14553 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:40.695852 14596 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.678747 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:40.679018 15153 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.697764 15153 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:40.699353 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1490) took 1249 ms. Trace:
I20260806 20:35:40.679435 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.684706 14550 raft_consensus.cc:2379] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:40.699808 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:40.664966 16893 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:40.689343 14473 rpcz_store.cc:276] 0806 20:35:39.332016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.560540 (+1228524us) service_pool.cc:227] Handling call
0806 20:35:40.568470 (+ 7930us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.568738 (+ 268us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:40.668343 (+ 99605us) log.cc:842] Serialized 14 byte log entry
0806 20:35:40.674251 (+ 5908us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.674295 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:40.674346 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.674405 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.674513 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":84968,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":23,"thread_start_us":100546,"threads_started":2}
I20260806 20:35:40.686158 14546 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:40.664848 16870 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:40.688889 16647 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:40.677654 15183 raft_consensus.cc:2396] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:40.692428 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1689) took 1585 ms. Trace:
I20260806 20:35:40.655026 16796 raft_consensus.cc:686] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:40.693686 14633 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 20:35:40.667810 14444 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:40.700335 14632 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.670148 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:40.701215 14554 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260806 20:35:40.764660 14518 connection.cc:232] RPC call timeout handler was delayed by -1.2049602689803578s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1506s for a response
W20260806 20:35:40.787885 14518 connection.cc:232] RPC call timeout handler was delayed by -1.1842602689866908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
W20260806 20:35:40.788414 14518 connection.cc:232] RPC call timeout handler was delayed by -1.1707602689857595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
I20260806 20:35:40.786782 14475 rpcz_store.cc:276] 0806 20:35:39.107460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.539115 (+1431655us) service_pool.cc:227] Handling call
0806 20:35:40.539435 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.539644 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:40.671767 (+132123us) log.cc:842] Serialized 14 byte log entry
0806 20:35:40.692117 (+ 20350us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.692162 (+ 45us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:40.692216 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.692271 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.692378 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":113900,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":68643,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":63,"thread_start_us":151839,"threads_started":2}
W20260806 20:35:40.788806 14518 connection.cc:232] RPC call timeout handler was delayed by -1.0708602689846884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
I20260806 20:35:40.772749 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1675) took 1988 ms. Trace:
W20260806 20:35:40.789614 14518 connection.cc:232] RPC call timeout handler was delayed by -1.0708602689846884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260806 20:35:40.790079 14518 connection.cc:232] RPC call timeout handler was delayed by -1.0114602689864114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
W20260806 20:35:40.790669 14518 connection.cc:232] RPC call timeout handler was delayed by -1.0051602689782158s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
I20260806 20:35:40.775211 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:40.792006 14548 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:40.685320 16823 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.747054 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1687) took 1640 ms. Trace:
I20260806 20:35:40.760154 16753 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:40.759295 16815 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:40.793903 16796 consensus_queue.cc:237] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.689188 16834 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:40.694303 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 548) took 1624 ms. Trace:
I20260806 20:35:40.794929 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.822502 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.823082 14625 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.823432 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:40.795171 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:40.799317 16786 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.800962 16647 raft_consensus.cc:686] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:40.805092 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 137353us
I20260806 20:35:40.805979 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:40.697203 16855 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:40.742853 14471 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:35:40.810220 14478 rpcz_store.cc:276] 0806 20:35:39.106779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.520492 (+1413713us) service_pool.cc:227] Handling call
0806 20:35:40.554590 (+ 34098us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.654699 (+100109us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.746751 (+ 92052us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b54001a4fd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:40.746978 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228916096}
I20260806 20:35:40.789837 14474 rpcz_store.cc:276] 0806 20:35:38.789453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.825077 (+1035624us) service_pool.cc:227] Handling call
0806 20:35:40.772502 (+947425us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54001a54d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:40.772675 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1866675456}
I20260806 20:35:40.831444 14444 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:40.814572 14476 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:40.806656 16870 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:40.811679 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 150865us
I20260806 20:35:40.830389 16910 raft_consensus.cc:2741] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260806 20:35:40.830199 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 134409us
I20260806 20:35:40.699828 14552 rpcz_store.cc:276] 0806 20:35:39.450160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.532831 (+1082671us) service_pool.cc:227] Handling call
0806 20:35:40.533110 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.533296 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:40.533574 (+ 278us) log.cc:842] Serialized 14 byte log entry
0806 20:35:40.698902 (+165328us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.698992 (+ 90us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:40.699060 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.699146 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.699284 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85538,"prepare.queue_time_us":2338,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":94291,"raft.run_wall_time_us":21516,"spinlock_wait_cycles":35858560,"thread_start_us":79403,"threads_started":1,"wal-append.queue_time_us":119044}
I20260806 20:35:40.686543 16545 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.837836 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1676) took 2036 ms. Trace:
W20260806 20:35:40.760249 16891 log.cc:924] Time spent T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260806 20:35:40.831800 14633 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:40.836259 14518 leader_election.cc:398] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5.
I20260806 20:35:40.842837 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.849696 14518 leader_election.cc:401] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:40.694077 15185 rpcz_store.cc:276] 0806 20:35:39.465328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.558383 (+1093055us) service_pool.cc:227] Handling call
0806 20:35:40.610316 (+ 51933us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.701078 16816 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:40.848443 15187 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:40.847787 14595 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:40.847618 14476 rpcz_store.cc:276] 0806 20:35:38.801236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:39.826011 (+1024775us) service_pool.cc:227] Handling call
0806 20:35:40.837762 (+1011751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.806689 14632 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:35:40.824290 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 539) took 1790 ms. Trace:
I20260806 20:35:40.817395 16893 raft_consensus.cc:686] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:40.763888 16683 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.853228 14478 raft_consensus.cc:1226] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:35:40.835105 16897 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:40.833353 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.834686 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:40.867745 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 211772us
I20260806 20:35:40.843991 16870 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.867195 14632 raft_consensus.cc:1226] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:40.899744 14476 raft_consensus.cc:1261] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 20:35:40.925915 14476 pending_rounds.cc:76] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 1
I20260806 20:35:40.926182 14476 pending_rounds.cc:97] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 20:35:40.926440 14476 raft_consensus.cc:2866] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:40.898530 16705 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5.
I20260806 20:35:40.842042 16903 raft_consensus.cc:2741] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:35:40.831398 16905 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:40.844426 16841 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:40.910146 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 284) took 1577 ms. Trace:
W20260806 20:35:40.911841 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 58500.3us
W20260806 20:35:40.898159 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 72288.4us
I20260806 20:35:40.865738 15183 rpcz_store.cc:276] 0806 20:35:39.027180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.435416 (+1408236us) service_pool.cc:227] Handling call
0806 20:35:40.817930 (+382514us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.909162 14596 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:40.918135 14625 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:35:40.919585 14442 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:40.854732 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.948348 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1486) took 1582 ms. Trace:
I20260806 20:35:40.948758 14555 rpcz_store.cc:276] 0806 20:35:39.365626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.431460 (+1065834us) service_pool.cc:227] Handling call
0806 20:35:40.431787 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.432482 (+ 695us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:40.432514 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:40.432995 (+ 481us) log.cc:842] Serialized 26 byte log entry
0806 20:35:40.487519 (+ 54524us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:40.487563 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.487639 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:40.948165 (+460526us) raft_consensus.cc:1671] finished
0806 20:35:40.948184 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.948290 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":137,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":54124,"threads_started":1,"wal-append.queue_time_us":85523}
I20260806 20:35:40.853154 16623 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:40.951269 16786 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.951735 14626 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:40.952636 14518 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:40.953028 15178 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:40.953552 15178 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 554) took 1881 ms. Trace:
W20260806 20:35:40.953640 16714 consensus_meta.cc:287] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.515s user 0.007s sys 0.004s
I20260806 20:35:40.953733 16873 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:40.822738 15182 rpcz_store.cc:276] 0806 20:35:39.070357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.591960 (+1521603us) service_pool.cc:227] Handling call
0806 20:35:40.592247 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.691805 (+ 99558us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.694220 (+ 2415us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:40.908272 16817 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:40.935037 14479 rpcz_store.cc:276] 0806 20:35:39.332323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.592705 (+1260382us) service_pool.cc:227] Handling call
0806 20:35:40.593006 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:40.593217 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:40.676659 (+ 83442us) log.cc:842] Serialized 14 byte log entry
0806 20:35:40.909675 (+233016us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:40.909746 (+ 71us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:40.909825 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.909925 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:40.910075 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":42441,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"thread_start_us":279767,"threads_started":2,"wal-append.queue_time_us":254065}
I20260806 20:35:40.945326 14624 raft_consensus.cc:2379] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:40.946995 14444 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:40.897498 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:40.955518 14442 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:40.955668 14633 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:40.959769 14549 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
W20260806 20:35:40.912935 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 101130us
I20260806 20:35:40.963110 14633 raft_consensus.cc:1226] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:40.964270 14633 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 7. Current term is 8. Ops: []
I20260806 20:35:40.934679 16841 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:40.966746 14441 leader_election.cc:334] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 3.697s (SENT)
I20260806 20:35:40.938107 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:40.954305 16873 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:40.965700 15186 raft_consensus.cc:2379] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:40.935586 16905 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:40.953953 15178 rpcz_store.cc:276] 0806 20:35:39.072538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.508203 (+1435665us) service_pool.cc:227] Handling call
0806 20:35:40.953482 (+445279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2432}
I20260806 20:35:40.968182 16624 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.937053 16854 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.970289 16817 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:40.954171 16714 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.909045 16834 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.987723 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.000025 14625 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:40.961679 16816 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:40.992209 14633 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:40.958971 16820 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:41.002709 14633 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:41.002882 14631 raft_consensus.cc:1226] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:40.990839 16873 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:40.936167 16647 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.960371 14518 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:40.988473 16853 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.969285 16888 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:40.973349 16841 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:40.986150 16753 raft_consensus.cc:686] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:40.990252 16822 consensus_queue.cc:1046] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260806 20:35:40.971673 16849 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:40.991161 16905 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:41.008889 14627 raft_consensus.cc:2354] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
I20260806 20:35:40.986398 16786 consensus_queue.cc:1046] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260806 20:35:40.993037 16815 raft_consensus.cc:686] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:41.040685 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 80349us
I20260806 20:35:41.041733 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:41.013139 16696 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:41.006707 16545 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:41.015412 16909 log.cc:924] Time spent T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260806 20:35:41.016806 16888 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:41.017517 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1683) took 1912 ms. Trace:
W20260806 20:35:41.019548 16894 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.327s user 0.001s sys 0.001s
I20260806 20:35:41.022730 14475 raft_consensus.cc:1226] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:41.003793 16816 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:41.021745 16753 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.038808 14625 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:41.006611 16851 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:40.961210 16893 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:41.047322 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 74696.4us
I20260806 20:35:41.014179 16834 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:41.050210 14628 raft_consensus.cc:2379] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.059547 14518 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:41.048404 14470 rpcz_store.cc:276] 0806 20:35:39.105292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.519794 (+1414502us) service_pool.cc:227] Handling call
0806 20:35:40.520095 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.569567 (+ 49472us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:40.569598 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:40.570045 (+ 447us) log.cc:842] Serialized 26 byte log entry
0806 20:35:40.692447 (+122402us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b18003c8b00. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a411 00007fb9b34c75cf
0806 20:35:40.956333 (+263886us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:40.956388 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:40.956502 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:41.017170 (+ 60668us) raft_consensus.cc:1671] finished
0806 20:35:41.017200 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:41.017451 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1557,"raft.queue_time_us":44460,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":31581,"spinlock_wait_cycles":27458816,"thread_start_us":263643,"threads_started":1,"wal-append.queue_time_us":408459}
W20260806 20:35:41.071169 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 162100us
I20260806 20:35:41.087893 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:41.097206 14518 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:41.007239 16286 consensus_queue.cc:1046] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260806 20:35:41.072607 15153 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:41.071602 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 223979us
I20260806 20:35:41.048087 16908 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:41.088255 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:41.023003 16853 consensus_queue.cc:1046] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260806 20:35:41.046456 16683 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:41.050037 16545 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:41.104822 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:41.201750 14632 raft_consensus.cc:1261] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:41.103741 14546 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:41.098714 14442 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:41.105293 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:41.285934 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 223953us
I20260806 20:35:41.104882 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:41.287539 14555 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:41.103518 14596 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:41.296052 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 192590us
I20260806 20:35:41.060434 16816 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:41.285355 14552 raft_consensus.cc:2362] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6. Re-sending same reply.
I20260806 20:35:41.215358 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 279) took 1884 ms. Trace:
I20260806 20:35:41.302071 14471 rpcz_store.cc:276] 0806 20:35:39.330706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.545497 (+1214791us) service_pool.cc:227] Handling call
0806 20:35:40.545809 (+ 312us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.841447 (+295638us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:41.215049 (+373602us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54003f61d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:41.215284 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":452527360}
W20260806 20:35:41.221540 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 64265.7us
I20260806 20:35:41.310902 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 285) took 1978 ms. Trace:
I20260806 20:35:41.311472 14475 rpcz_store.cc:276] 0806 20:35:39.332688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.789312 (+1456624us) service_pool.cc:227] Handling call
0806 20:35:40.795891 (+ 6579us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:41.052496 (+256605us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:41.310608 (+258112us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54001068d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:41.310828 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":521256320}
W20260806 20:35:41.160200 16907 log.cc:924] Time spent T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.225s user 0.001s sys 0.000s
W20260806 20:35:41.321477 15150 leader_election.cc:334] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.535s (SENT)
I20260806 20:35:41.263659 14624 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
W20260806 20:35:41.262212 16818 consensus_meta.cc:287] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.824s user 0.011s sys 0.000s
I20260806 20:35:41.058485 16897 raft_consensus.cc:686] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:35:41.090258 16872 consensus_meta.cc:287] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.505s user 0.014s sys 0.001s
I20260806 20:35:41.299592 14549 raft_consensus.cc:2379] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.060508 16696 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:41.326176 14632 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:41.273917 15183 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:41.142316 16796 raft_consensus.cc:2741] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:35:41.097976 16524 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:41.329226 14628 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 20:35:41.172945 16884 consensus_queue.cc:1035] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.193s
W20260806 20:35:41.332968 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 292540us
I20260806 20:35:41.259112 16624 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:41.097411 16714 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:41.100873 16623 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:35:41.274210 16855 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:41.266176 16854 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:41.137122 16817 raft_consensus.cc:686] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:41.265173 16903 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260806 20:35:41.070175 16888 consensus_queue.cc:1046] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.255s
I20260806 20:35:41.316900 16683 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:41.346220 14627 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:41.342752 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 338607us
I20260806 20:35:41.347141 14595 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.309753 16286 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:35:41.296926 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 198298us
I20260806 20:35:41.317286 16705 consensus_queue.cc:1046] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20260806 20:35:41.162734 16654 consensus_queue.cc:1035] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.180s
I20260806 20:35:41.143819 16815 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:41.564687 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 217242us
W20260806 20:35:41.498656 15150 leader_election.cc:334] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.571s (SENT)
W20260806 20:35:41.565846 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 67251.7us
W20260806 20:35:41.397385 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 420172us
I20260806 20:35:41.568279 14626 raft_consensus.cc:1226] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
W20260806 20:35:41.569139 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 222129us
I20260806 20:35:41.569506 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:41.568274 14518 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.496959 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:41.101331 16917 raft_consensus.cc:3032] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:41.351370 14547 raft_consensus.cc:1261] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:41.570513 14550 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:41.570986 14546 raft_consensus.cc:2379] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:41.569836 14595 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.559964 14521 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:41.576177 14359 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 7f41753baced49a1997940dfc28b2e00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica df1245ecd2034e11ae6cf688c1c10a4d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
W20260806 20:35:41.497157 16906 log.cc:924] Time spent T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.575s user 0.004s sys 0.000s
I20260806 20:35:41.340921 16849 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:41.486217 16786 raft_consensus.cc:3032] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 20:35:41.522262 16907 log.cc:924] Time spent T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260806 20:35:41.600244 15150 leader_election.cc:334] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.669s (SENT)
I20260806 20:35:41.343458 16696 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:41.604370 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 44516.5us
I20260806 20:35:41.628496 14521 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:41.629853 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 57584.9us
W20260806 20:35:41.541467 16870 consensus_meta.cc:287] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.606s user 0.003s sys 0.007s
I20260806 20:35:41.606889 16903 consensus_queue.cc:1225] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer 8e0cfd35f2994f91aaae46c25c6d6bff log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:35:41.612325 14624 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 20:35:41.523905 16820 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:41.535392 16844 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:41.569244 16524 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:41.556044 16930 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:41.575639 16855 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:41.571929 16917 raft_consensus.cc:729] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:41.631811 14444 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:41.572261 16852 consensus_queue.cc:1046] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.175s
I20260806 20:35:41.613945 16786 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:41.579134 16545 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:41.775619 16524 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:41.787087 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 155393us
I20260806 20:35:41.789501 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:41.680763 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 49852.1us
I20260806 20:35:41.351065 16848 consensus_queue.cc:1046] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.277s
I20260806 20:35:41.672629 16654 consensus_queue.cc:1046] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:41.608525 16888 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:41.631479 16696 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:41.780223 16937 raft_consensus.cc:3032] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 20:35:41.694828 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 122698us
W20260806 20:35:41.720227 15150 leader_election.cc:334] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.711s (SENT)
W20260806 20:35:41.630975 16834 consensus_meta.cc:287] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.535s user 0.012s sys 0.000s
I20260806 20:35:41.199368 16921 consensus_queue.cc:1035] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.424s
I20260806 20:35:41.577162 16856 consensus_queue.cc:1035] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260806 20:35:41.345952 16841 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.336796 16818 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.607702 16683 raft_consensus.cc:686] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:41.797415 16937 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:41.493829 16905 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.824827 16937 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.814644 16834 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:35:41.826295 16929 log.cc:924] Time spent T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.567s user 0.003s sys 0.000s
W20260806 20:35:41.812201 16934 log.cc:924] Time spent T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20260806 20:35:41.827557 16905 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:41.627518 16875 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:41.702278 16705 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:41.777621 14595 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:41.704433 16286 raft_consensus.cc:686] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:41.775637 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 145370us
I20260806 20:35:41.778952 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1758) took 1160 ms. Trace:
I20260806 20:35:41.540750 16897 consensus_queue.cc:237] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.635082 16853 consensus_queue.cc:1046] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.122s
I20260806 20:35:41.596365 16872 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:41.666389 16870 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.760250 16820 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:41.630540 16926 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.164s
W20260806 20:35:41.664585 16936 log.cc:924] Time spent T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260806 20:35:41.790608 14518 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:41.866375 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 75868.3us
I20260806 20:35:41.867341 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.791344 14630 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.795121 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:41.604007 16714 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:41.796818 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:41.871351 14627 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:41.798866 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 78724.8us
I20260806 20:35:41.794590 16915 raft_consensus.cc:3032] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:35:41.792204 16624 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:41.816190 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:41.878582 15184 raft_consensus.cc:2379] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:41.879200 14628 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.808917 16888 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.822484 16818 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:41.823523 16683 consensus_queue.cc:237] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.891434 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.892344 14630 raft_consensus.cc:2379] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.840233 14477 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:41.779459 16844 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.898134 16844 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:41.843672 14629 rpcz_store.cc:276] 0806 20:35:40.618747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.618884 (+ 137us) service_pool.cc:227] Handling call
0806 20:35:40.668368 (+ 49484us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:40.674873 (+ 6505us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:40.674906 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:40.675510 (+ 604us) log.cc:842] Serialized 26 byte log entry
0806 20:35:40.811340 (+135830us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:40.832077 (+ 20737us) log.cc:842] Serialized 14 byte log entry
0806 20:35:41.044143 (+212066us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b1800211300. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05424ca 00007fb9b9c23f99 00007fb9b0406990 00007fb9b05c8864 00007fb9b05c8155 00007fb9b34eb3d0 00007fb9b34fd576 00007fb9b350e0d7 00007fb9b350e058
0806 20:35:41.044309 (+ 166us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:41.044378 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:41.765620 (+721242us) raft_consensus.cc:1671] finished
0806 20:35:41.778573 (+ 12953us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:41.778866 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5465,"prepare.queue_time_us":24817,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":3365,"raft.queue_time_us":378738,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":3684,"spinlock_wait_cycles":8669440,"thread_start_us":32748,"threads_started":2,"wal-append.queue_time_us":188983}
I20260806 20:35:41.899437 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.900280 14629 raft_consensus.cc:2379] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.876627 16915 raft_consensus.cc:729] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:41.901196 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:41.902518 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.851776 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:41.854739 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 77269.2us
I20260806 20:35:41.893646 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.855556 14521 leader_election.cc:398] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8.
W20260806 20:35:41.831130 16925 log.cc:924] Time spent T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.274s user 0.000s sys 0.001s
I20260806 20:35:41.861713 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:41.886178 16873 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.875s user 0.006s sys 0.005s
I20260806 20:35:41.826310 16937 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:41.869244 14444 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:41.871996 14518 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:41.872825 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 67468.8us
I20260806 20:35:41.781695 16817 consensus_queue.cc:237] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.820641 16943 consensus_queue.cc:1046] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260806 20:35:41.863463 16823 consensus_meta.cc:287] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.030s user 0.008s sys 0.005s
I20260806 20:35:41.862814 16905 raft_consensus.cc:2741] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 20:35:41.884824 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.785844 16786 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.855207 16872 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:41.756063 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 142847us
W20260806 20:35:41.894088 16916 log.cc:924] Time spent T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.787s user 0.000s sys 0.003s
I20260806 20:35:41.893587 16286 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.894145 16888 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:41.894703 16927 log.cc:924] Time spent T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.317s user 0.001s sys 0.000s
W20260806 20:35:41.895112 16938 log.cc:924] Time spent T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260806 20:35:41.895677 16941 raft_consensus.cc:3032] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:41.895898 16841 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:41.904141 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.905790 14521 leader_election.cc:401] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:41.903303 14629 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.818394 16654 raft_consensus.cc:3032] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:41.876910 16705 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:41.915313 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:41.915611 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 43729.2us
I20260806 20:35:41.933125 16872 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:41.925259 14550 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:41.962625 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:41.929373 14626 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.962723 14627 raft_consensus.cc:2379] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.901602 16915 consensus_queue.cc:260] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.933842 14625 raft_consensus.cc:2379] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:41.934942 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 65827.4us
I20260806 20:35:41.978763 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.979737 14626 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.935519 14471 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 7. Current term is 8. Ops: []
I20260806 20:35:41.936008 14595 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.937402 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:41.937635 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:41.937793 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:41.980142 14444 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:41.938124 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:41.939594 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1680) took 2925 ms. Trace:
I20260806 20:35:41.939857 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1691) took 1347 ms. Trace:
I20260806 20:35:41.941109 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:41.941577 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.914348 16853 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:41.947016 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:41.887239 16624 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.851083 16917 consensus_queue.cc:260] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.928937 16823 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.920745 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.998149 14630 raft_consensus.cc:2379] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.999012 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:41.999905 14630 raft_consensus.cc:2379] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.966303 14518 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.967288 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:41.948565 16753 consensus_queue.cc:1046] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20260806 20:35:42.001263 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 65365.9us
I20260806 20:35:41.958381 16943 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:42.003332 16943 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.004729 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:42.005525 14476 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:42.006387 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:42.006592 16943 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.007424 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:42.007529 16933 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20260806 20:35:42.008047 14476 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:42.008945 14444 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.003369 16951 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:42.010211 14444 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.011328 14444 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:41.961122 16647 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:42.012172 16935 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:42.012619 16942 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.012434 16647 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:41.929003 16924 consensus_queue.cc:1046] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20260806 20:35:42.015127 16924 raft_consensus.cc:3032] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:42.015479 16924 raft_consensus.cc:729] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:41.983922 14477 rpcz_store.cc:276] 0806 20:35:39.032673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.429345 (+1396672us) service_pool.cc:227] Handling call
0806 20:35:40.429658 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:41.892851 (+1463193us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:41.939327 (+ 46476us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400333bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:41.939507 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3220565760}
I20260806 20:35:42.016234 16924 consensus_queue.cc:260] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.017647 16924 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.017479 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:42.018584 14477 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:41.942150 16844 raft_consensus.cc:3032] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 20:35:41.990556 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 147823us
I20260806 20:35:41.984879 14628 raft_consensus.cc:2379] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:41.976508 16875 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:41.922907 16873 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.989867 14470 raft_consensus.cc:2362] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6. Re-sending same reply.
I20260806 20:35:41.963549 16854 raft_consensus.cc:3032] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:41.950984 16941 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:41.997943 15151 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:42.012732 16926 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:42.012925 16935 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:42.041607 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:42.019434 14521 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.959175 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1625) took 2569 ms. Trace:
I20260806 20:35:41.990553 14632 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:42.054153 14547 rpcz_store.cc:276] 0806 20:35:39.379892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.498859 (+1118967us) service_pool.cc:227] Handling call
0806 20:35:40.499155 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:41.572550 (+1073395us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:41.948427 (+375877us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54000a8a54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:41.949217 (+ 790us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1954230016}
I20260806 20:35:42.055166 16705 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:42.055964 16705 raft_consensus.cc:686] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:42.057253 16705 consensus_queue.cc:237] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:41.992753 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:42.060022 15187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:35:42.060629 15187 raft_consensus.cc:595] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:42.061008 15187 raft_consensus.cc:603] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:42.061163 14626 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:42.022984 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:42.064002 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 66157.9us
I20260806 20:35:41.959735 16654 raft_consensus.cc:729] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:42.070514 14476 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:42.024624 14627 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:42.025374 16844 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:41.961272 16945 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.027710 14470 raft_consensus.cc:1226] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: []
W20260806 20:35:42.027570 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 61376.8us
I20260806 20:35:42.037503 16848 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:42.029145 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:42.029467 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.030841 16624 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:42.031057 16915 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:42.000429 16947 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.031715 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:42.092912 14473 raft_consensus.cc:2379] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:42.032639 16903 raft_consensus.cc:3032] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:42.034005 16873 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.034905 16852 raft_consensus.cc:3032] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:42.020311 16853 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.035652 14474 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:42.036571 16823 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.016127 16647 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.040370 14631 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 20:35:42.067148 14628 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:42.115089 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:42.115731 14628 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:42.021083 16714 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.064983 14444 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.117419 14549 raft_consensus.cc:1261] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:35:42.020366 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:42.118616 14629 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:42.118700 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:42.074939 15153 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:42.081895 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:41.984014 14469 rpcz_store.cc:276] 0806 20:35:40.592258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.825504 (+233246us) service_pool.cc:227] Handling call
0806 20:35:40.846887 (+ 21383us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:41.082192 (+235305us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:41.082239 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:41.082875 (+ 636us) log.cc:842] Serialized 26 byte log entry
0806 20:35:41.211114 (+128239us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:41.258562 (+ 47448us) log.cc:842] Serialized 14 byte log entry
0806 20:35:41.258790 (+ 228us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:41.317326 (+ 58536us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b54002170d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:41.317370 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:41.939583 (+622213us) raft_consensus.cc:1671] finished
0806 20:35:41.939681 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:41.939803 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14379,"prepare.queue_time_us":48040,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":357441,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":6217,"spinlock_wait_cycles":45819648,"thread_start_us":341042,"threads_started":2,"wal-append.queue_time_us":265458}
W20260806 20:35:42.119678 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 54799.5us
I20260806 20:35:42.081406 16943 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260806 20:35:42.038132 16854 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:42.084172 14471 raft_consensus.cc:1261] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 20:35:42.086108 15150 leader_election.cc:334] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 2.132s (SENT)
I20260806 20:35:42.086009 16941 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:42.088748 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 66460.9us
I20260806 20:35:42.096120 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:42.095789 16903 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:42.097502 14521 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:42.162143 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 64731us
I20260806 20:35:42.162652 14521 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:42.077411 16908 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:42.083762 16753 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:35:42.103992 15151 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:42.167351 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:42.167867 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 63958.4us
I20260806 20:35:42.054795 16893 consensus_queue.cc:1046] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260806 20:35:42.120666 14444 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:42.210312 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 89725.5us
I20260806 20:35:42.211566 14444 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.083257 14632 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:42.122004 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47185.5us
I20260806 20:35:42.097165 14627 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:42.156225 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.156397 15150 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:42.138635 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.157649 14477 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:42.121167 14596 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:42.215512 14553 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:42.216071 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 94986.2us
W20260806 20:35:42.222129 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 136075us
I20260806 20:35:42.159734 16647 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:42.256979 16624 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:35:42.248610 16905 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:42.258916 16955 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:42.259128 16905 raft_consensus.cc:686] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:42.266059 14471 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.126881 14628 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:35:42.160239 15184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 562) took 1351 ms. Trace:
I20260806 20:35:42.160974 14547 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:42.163045 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.268173 15184 rpcz_store.cc:276] 0806 20:35:40.808407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.808554 (+ 147us) service_pool.cc:227] Handling call
0806 20:35:42.160078 (+1351524us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54003d0ed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:42.160192 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2422841984}
I20260806 20:35:42.088827 16856 consensus_queue.cc:1046] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260806 20:35:42.159781 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.166448 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.279844 14549 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:42.119617 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:42.101708 16852 raft_consensus.cc:729] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:42.169287 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 87501.1us
I20260806 20:35:42.100129 16870 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.099466 16930 raft_consensus.cc:686] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:42.299649 16930 consensus_queue.cc:237] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.304754 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:42.305640 14477 raft_consensus.cc:2379] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:42.170351 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:42.122813 16823 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:35:42.195062 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:42.201027 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.322170 14469 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:42.322788 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.323480 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.324059 14555 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.201774 16875 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:42.202669 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.202963 16917 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.326028 14548 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.170996 14629 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:42.204115 14625 raft_consensus.cc:1226] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 7. Current term is 8. Ops: []
I20260806 20:35:42.329658 16856 consensus_queue.cc:1046] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:42.331812 16852 consensus_queue.cc:260] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.204272 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.204918 14595 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:42.209097 15151 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:42.333268 16852 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.223930 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.333523 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.333758 14548 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:42.334007 14552 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.334163 14551 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.225539 16908 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:42.227694 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1697) took 1615 ms. Trace:
I20260806 20:35:42.337072 14474 rpcz_store.cc:276] 0806 20:35:40.612628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.837020 (+224392us) service_pool.cc:227] Handling call
0806 20:35:40.837328 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:42.114028 (+1276700us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:42.227439 (+113411us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003f4dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:42.227630 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2755395840}
I20260806 20:35:42.229342 16884 raft_consensus.cc:3032] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
W20260806 20:35:42.333362 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 128532us
I20260806 20:35:42.338064 16884 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:42.338866 16884 consensus_queue.cc:260] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.339854 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:42.232338 16714 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:42.233428 16816 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.902s user 0.011s sys 0.000s
I20260806 20:35:42.340819 16816 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.342762 16816 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.343286 14624 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.231645 16853 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:42.344439 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.285655 16903 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.345783 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.346065 16903 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.233767 15187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:35:42.235471 16945 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:42.348729 15187 raft_consensus.cc:595] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:42.213336 16796 raft_consensus.cc:3032] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:42.349043 16945 raft_consensus.cc:3032] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:42.349376 16796 raft_consensus.cc:729] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:42.349565 16945 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:42.348886 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:42.213559 16933 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260806 20:35:42.350729 14548 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:42.170611 16753 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:42.244243 16854 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.244946 16962 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:42.213559 16922 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:42.132189 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:42.267421 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 47052.9us
I20260806 20:35:42.270661 14632 raft_consensus.cc:1226] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:42.371140 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.271126 16848 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:42.270865 16855 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.250061 16647 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:42.264663 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60780 (request call id 393) took 1408 ms. Trace:
I20260806 20:35:42.668617 15183 rpcz_store.cc:276] 0806 20:35:40.814344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:40.960992 (+146648us) service_pool.cc:227] Handling call
0806 20:35:40.994178 (+ 33186us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:41.351274 (+357096us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:42.171615 (+820341us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54004aef54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:42.222301 (+ 50686us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":642721536}
I20260806 20:35:42.661855 14471 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:42.272620 16926 consensus_queue.cc:1046] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:42.259243 16942 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:42.273748 16786 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.275569 16941 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.276013 16910 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:42.278658 16951 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8.
I20260806 20:35:42.283036 14553 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:42.290570 14442 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.333182 16624 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:35:42.223060 15150 leader_election.cc:334] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 2.171s (SENT)
W20260806 20:35:42.681866 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 458808us
W20260806 20:35:42.682689 15150 leader_election.cc:334] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 2.178s (SENT)
I20260806 20:35:42.683734 16943 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:42.340040 16884 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.350329 16945 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.684268 16943 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.685115 16945 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.686482 16943 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.220485 16946 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:42.688660 16946 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:42.689038 16946 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.371717 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.349103 15187 raft_consensus.cc:603] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:42.374197 14549 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:42.375289 14479 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:42.351980 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.695612 14547 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.324007 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.704185 16946 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.711063 16946 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:42.379106 14624 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:42.379899 14551 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:42.715173 14551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:42.715795 14551 raft_consensus.cc:595] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.716111 14551 raft_consensus.cc:603] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:42.450701 16822 raft_consensus.cc:3032] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:42.718052 16822 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:42.718365 14596 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:42.489562 16937 consensus_meta.cc:287] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.569s user 0.009s sys 0.000s
W20260806 20:35:42.440757 16696 consensus_meta.cc:287] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.624s user 0.003s sys 0.007s
I20260806 20:35:42.719338 16624 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:42.719861 16624 raft_consensus.cc:686] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:42.719597 16696 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.720773 16624 consensus_queue.cc:237] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.721609 16696 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:42.660887 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 498250us
W20260806 20:35:42.439996 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 158569us
I20260806 20:35:42.437745 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:42.460618 16820 consensus_meta.cc:287] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.588s user 0.009s sys 0.002s
I20260806 20:35:42.728963 16820 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.729473 14546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:42.730136 14546 raft_consensus.cc:595] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.730504 14546 raft_consensus.cc:603] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:42.433605 16955 consensus_queue.cc:1046] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.232s
I20260806 20:35:42.409806 16647 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.732071 16955 raft_consensus.cc:3032] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:42.438776 16854 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.734589 14521 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.121132 16844 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.109375 16654 consensus_queue.cc:260] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.412832 16286 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.254s
I20260806 20:35:42.481211 14626 raft_consensus.cc:1226] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 20:35:42.412308 14555 raft_consensus.cc:2396] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:42.739403 14551 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:42.739157 16647 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.429097 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1769) took 1622 ms. Trace:
I20260806 20:35:42.740177 14547 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:42.740480 16945 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:42.351266 16753 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:42.740248 14633 rpcz_store.cc:276] 0806 20:35:40.806790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:41.008622 (+201832us) service_pool.cc:227] Handling call
0806 20:35:41.020859 (+ 12237us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:35:41.072026 (+ 51167us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:41.072074 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 20:35:41.107137 (+ 35063us) log.cc:842] Serialized 78 byte log entry
0806 20:35:41.428700 (+321563us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:41.608412 (+179712us) log.cc:842] Serialized 14 byte log entry
0806 20:35:41.621537 (+ 13125us) log.cc:842] Serialized 14 byte log entry
0806 20:35:41.621717 (+ 180us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:41.621809 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:42.034194 (+412385us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b180005c800. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:42.428772 (+394578us) raft_consensus.cc:1671] finished
0806 20:35:42.428802 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:42.429018 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62137,"prepare.queue_time_us":280803,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":98,"raft.queue_time_us":74518,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":16766,"spinlock_wait_cycles":7774592,"thread_start_us":265774,"threads_started":1,"wal-append.queue_time_us":509599}
W20260806 20:35:42.666785 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 457715us
I20260806 20:35:42.741271 16647 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:42.386596 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 129865us
I20260806 20:35:42.736038 14521 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.711113 14550 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:35:42.376370 14552 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:42.731267 16820 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:42.417101 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 69719.8us
I20260806 20:35:42.761842 14552 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:42.766224 14596 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:42.768627 14521 leader_election.cc:398] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
I20260806 20:35:42.409861 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:42.769084 14521 leader_election.cc:401] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 20:35:42.770512 14548 raft_consensus.cc:2396] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:42.771791 14596 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:42.774572 14552 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260806 20:35:42.775915 14552 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 20:35:42.643879 16915 consensus_meta.cc:287] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.541s user 0.012s sys 0.000s
W20260806 20:35:42.782011 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 410353us
I20260806 20:35:42.784308 14479 raft_consensus.cc:1261] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:35:42.786322 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 495837us
I20260806 20:35:42.784802 14474 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260806 20:35:42.636793 16872 consensus_meta.cc:287] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.643s user 0.006s sys 0.006s
I20260806 20:35:42.788976 14476 raft_consensus.cc:1226] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:35:42.789971 14469 raft_consensus.cc:1261] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:42.790102 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:42.790834 14476 raft_consensus.cc:2379] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:42.789099 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:42.507736 16524 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.710s user 0.009s sys 0.002s
I20260806 20:35:42.793608 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:42.513042 16875 raft_consensus.cc:686] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:42.809161 16974 consensus_queue.cc:1035] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:35:42.809458 16822 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.809839 16875 consensus_queue.cc:237] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.810509 16822 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.811004 16974 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:42.813088 16965 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:42.813803 16965 consensus_queue.cc:1035] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:35:42.815621 16965 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
I20260806 20:35:42.817322 16654 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:42.795540 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 79806.6us
I20260806 20:35:42.413605 16962 raft_consensus.cc:686] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:42.819926 15150 leader_election.cc:334] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 2.181s (SENT)
I20260806 20:35:42.820237 16962 consensus_queue.cc:237] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:42.821596 15150 leader_election.cc:334] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 2.214s (SENT)
I20260806 20:35:42.372145 16714 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:42.822453 16714 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.732501 16955 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:42.823630 16844 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.797221 15187 raft_consensus.cc:2354] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4.
I20260806 20:35:42.798182 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:42.825323 15184 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:42.794538 16820 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:42.799700 14473 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:35:42.800688 14474 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:42.387130 16897 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:42.769852 16683 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:42.386340 16905 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.827384 16683 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.830148 16683 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:42.770841 16893 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 20:35:42.785429 16903 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:42.785799 16910 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:42.439666 16956 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:42.808210 16834 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:42.849677 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 62264us
I20260806 20:35:42.749759 16943 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260806 20:35:42.821759 16817 raft_consensus.cc:3032] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:42.824182 16955 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.784787 16870 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260806 20:35:42.801615 16972 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:35:42.831106 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 45855.6us
I20260806 20:35:42.505666 16947 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:42.895440 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:42.827116 16714 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:42.851248 14595 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:42.866113 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:42.869243 14442 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.873231 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:42.874039 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 82757us
I20260806 20:35:42.876531 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:42.904752 14548 raft_consensus.cc:595] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.409888 16853 consensus_queue.cc:1046] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.462s
I20260806 20:35:42.880851 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:42.889986 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:42.913910 14626 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:42.914348 15187 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:42.893698 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:42.350267 16796 consensus_queue.cc:260] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.498435 16855 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:42.899622 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:42.905514 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:42.921418 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 173029us
I20260806 20:35:42.874430 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:42.903110 15184 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:42.915110 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 45954.6us
I20260806 20:35:42.943377 14442 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.915119 14550 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.913704 14479 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:42.878746 16817 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:42.796183 16872 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:42.906301 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:42.798038 16524 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.881376 16956 raft_consensus.cc:686] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:42.981783 14550 raft_consensus.cc:2396] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:35:42.890806 16910 raft_consensus.cc:686] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:35:42.982970 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 66644.9us
I20260806 20:35:42.983474 14624 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:42.984124 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:42.985020 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:42.881003 16822 consensus_queue.cc:1046] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260806 20:35:42.874665 16903 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:42.881539 16971 consensus_queue.cc:1046] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.153s
I20260806 20:35:42.901707 16917 consensus_queue.cc:1035] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.128s
I20260806 20:35:42.905651 16856 consensus_queue.cc:1046] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.161s
I20260806 20:35:43.122224 14633 raft_consensus.cc:1226] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:42.987030 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:35:43.130733 14629 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:43.026986 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 83717.4us
I20260806 20:35:43.052894 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.132256 14442 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:42.985795 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:43.133213 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 282027us
I20260806 20:35:43.134227 14595 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:43.132391 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:35:43.133677 14631 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.052588 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:42.905623 16972 raft_consensus.cc:686] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:43.135438 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:43.135810 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.081334 15185 raft_consensus.cc:2354] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8.
I20260806 20:35:43.165803 14442 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.167100 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.167217 14442 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:43.168396 14442 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:43.156982 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 174162us
I20260806 20:35:43.173364 14624 raft_consensus.cc:2396] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:42.942324 16624 raft_consensus.cc:2741] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 20:35:43.170783 16942 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.017899 16796 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:42.914556 16965 consensus_queue.cc:1046] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.250s
I20260806 20:35:43.188822 14630 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 20:35:43.184137 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:43.190390 14633 raft_consensus.cc:2379] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.191763 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.192484 14625 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:43.122059 14444 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:43.080188 15178 raft_consensus.cc:2396] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:43.198489 14547 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:43.198570 15153 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:43.174568 16905 raft_consensus.cc:2741] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:35:42.943984 16935 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.171519 14546 raft_consensus.cc:1226] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:43.063771 16854 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 20:35:43.177982 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 45163.8us
I20260806 20:35:43.180986 14626 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 20:35:43.181134 15151 leader_election.cc:334] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.596s (SENT)
I20260806 20:35:43.184170 14553 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:43.157687 14479 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:43.206260 15151 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.158839 14632 raft_consensus.cc:2379] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:43.186139 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:43.145632 16823 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:42.917557 16951 consensus_queue.cc:1035] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.258s
I20260806 20:35:43.171413 16955 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:43.195153 16910 consensus_queue.cc:237] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:43.216785 15151 leader_election.cc:334] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.070s (SENT)
I20260806 20:35:43.217150 15151 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.182868 16852 raft_consensus.cc:3032] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
W20260806 20:35:43.197144 16977 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260806 20:35:43.196812 16972 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.191720 16884 consensus_queue.cc:1035] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260806 20:35:43.174110 16908 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:43.174952 16816 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.170761 16820 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:35:43.156397 15187 raft_consensus.cc:2362] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6. Re-sending same reply.
I20260806 20:35:43.052910 16962 consensus_queue.cc:1046] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20260806 20:35:43.163933 14554 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260806 20:35:43.121308 16975 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
W20260806 20:35:42.995441 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 226999us
W20260806 20:35:43.200281 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 78320.6us
I20260806 20:35:43.200428 14442 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:43.200865 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.192945 16817 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.208104 14628 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:43.248878 16817 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.223882 16971 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260806 20:35:43.118937 16753 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.252717 16696 consensus_queue.cc:1046] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:43.254057 16753 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:43.188207 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.254281 16986 consensus_queue.cc:1046] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:43.255182 15182 raft_consensus.cc:2396] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 20:35:43.216157 15186 raft_consensus.cc:2354] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
I20260806 20:35:43.206283 16853 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:43.256132 16696 raft_consensus.cc:3032] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:43.205076 16286 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:43.256822 16286 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:43.256979 15183 raft_consensus.cc:2379] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 2: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:43.257262 16286 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:43.192263 16980 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.228649 16844 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:43.267457 14628 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
W20260806 20:35:43.233922 15151 leader_election.cc:334] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.158s (SENT)
I20260806 20:35:43.219885 16852 raft_consensus.cc:729] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:43.240203 14470 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:43.253890 16873 consensus_queue.cc:1046] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:43.256171 16969 raft_consensus.cc:979] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff: : Instructing follower df1245ecd2034e11ae6cf688c1c10a4d to start an election
I20260806 20:35:43.222720 14518 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:43.205751 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:43.271286 16853 raft_consensus.cc:686] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:43.281395 16853 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.207599 16897 raft_consensus.cc:2741] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:35:43.258688 16872 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:43.213595 16815 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:43.259136 16696 raft_consensus.cc:729] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:43.270864 15153 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:43.271966 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 71181us
I20260806 20:35:43.293926 14595 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:43.272771 14549 raft_consensus.cc:1261] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:43.278846 15180 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:43.235126 16908 raft_consensus.cc:686] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:43.259670 16962 consensus_queue.cc:1225] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer 92828dd6dbe74f29a04d4e702b502621 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 20:35:43.313501 14625 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:43.290333 16971 raft_consensus.cc:3032] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
W20260806 20:35:43.268239 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 67894.6us
I20260806 20:35:43.220640 16954 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260806 20:35:43.272346 16974 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:43.276407 16816 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 20:35:43.297238 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 74596.2us
I20260806 20:35:43.301772 16873 raft_consensus.cc:1067] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Signalling peer df1245ecd2034e11ae6cf688c1c10a4d to start an election
I20260806 20:35:43.300096 14444 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.299937 16823 consensus_queue.cc:1046] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:35:43.304158 14549 raft_consensus.cc:1261] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:43.304425 14626 raft_consensus.cc:1261] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:35:43.306175 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:43.287868 14552 raft_consensus.cc:1261] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:35:43.317739 14442 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.290197 16903 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.322860 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.334492 14444 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.335098 14476 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.297433 16915 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.325168 14550 raft_consensus.cc:1226] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:43.335672 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:43.336267 14552 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:43.299705 14628 raft_consensus.cc:1261] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:43.339233 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.326854 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:43.341605 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.342631 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:43.342888 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.343597 14479 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.344024 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.306784 16815 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:43.315721 16986 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:43.351586 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.313831 16524 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.342159 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.354517 14474 raft_consensus.cc:2379] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.356379 15184 raft_consensus.cc:2396] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:35:43.355445 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.352108 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.357714 14476 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.356721 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.228811 16956 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.359552 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.271169 16980 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.320864 16971 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:43.321488 16974 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:43.359717 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.363961 14631 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:43.365217 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.374627 14476 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.346800 16854 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:43.373039 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.349979 16926 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:43.332954 16973 raft_consensus.cc:2741] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260806 20:35:43.369525 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:43.343763 16935 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:43.365873 14625 raft_consensus.cc:1261] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:35:43.353505 16753 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260806 20:35:43.352499 16930 consensus_queue.cc:1046] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260806 20:35:43.352591 16942 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:43.378885 14474 raft_consensus.cc:2379] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.366712 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.396836 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.399735 14629 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:43.380023 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.380295 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:43.384680 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47877.4us
I20260806 20:35:43.351696 16816 raft_consensus.cc:686] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:43.401150 14479 raft_consensus.cc:2379] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.385704 14476 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:43.388936 14444 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:43.393409 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.362388 16815 raft_consensus.cc:686] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:43.404194 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.404860 14473 raft_consensus.cc:2379] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:43.405246 14474 raft_consensus.cc:2379] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.405522 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.403191 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.377523 16974 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.406795 14478 raft_consensus.cc:2379] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.407548 14476 raft_consensus.cc:2379] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.405292 14469 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.350209 16915 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.333112 16905 consensus_queue.cc:1035] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260806 20:35:43.333820 16852 consensus_queue.cc:260] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.412021 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.414290 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:43.414861 16984 log.cc:924] Time spent T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260806 20:35:43.414652 16816 consensus_queue.cc:237] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.445405 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:43.415686 16994 log.cc:924] Time spent T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
W20260806 20:35:43.416390 16992 log.cc:924] Time spent T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260806 20:35:43.416719 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.400565 16820 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:43.401150 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.421761 14631 raft_consensus.cc:1261] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:43.380926 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:43.443859 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.369520 16696 consensus_queue.cc:260] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.450800 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.453359 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.457695 16980 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.502676 14469 raft_consensus.cc:2379] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.458148 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1512) took 1137 ms. Trace:
I20260806 20:35:43.460039 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.461093 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1509) took 1174 ms. Trace:
I20260806 20:35:43.503199 14547 rpcz_store.cc:276] 0806 20:35:42.321111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.740969 (+419858us) service_pool.cc:227] Handling call
0806 20:35:42.741326 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:43.199036 (+457710us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:43.457895 (+258859us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54002da5d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:43.458084 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1050832640}
I20260806 20:35:43.503352 14546 rpcz_store.cc:276] 0806 20:35:42.287091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.731133 (+444042us) service_pool.cc:227] Handling call
0806 20:35:42.731453 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:43.208784 (+477331us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:43.460824 (+252040us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54002dafd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:43.461028 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1011645696}
I20260806 20:35:43.466126 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:43.378363 16908 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:43.469947 16995 log.cc:924] Time spent T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
W20260806 20:35:43.470709 16997 log.cc:924] Time spent T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260806 20:35:43.471216 16815 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.410681 16935 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:43.387656 16624 consensus_queue.cc:1035] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260806 20:35:43.479221 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.438895 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:43.481792 16982 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.202s user 0.001s sys 0.001s
W20260806 20:35:43.515715 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 179042us
W20260806 20:35:43.500522 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 229756us
I20260806 20:35:43.502068 14478 raft_consensus.cc:2362] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7. Re-sending same reply.
I20260806 20:35:43.408427 16942 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.516991 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:43.522775 14546 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
I20260806 20:35:43.523432 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:43.523797 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:43.524209 14547 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:43.524356 14546 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:43.525009 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:43.525594 14547 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:43.526322 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:43.526818 14547 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:43.527838 15150 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.422646 16971 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:43.529313 16991 log.cc:924] Time spent T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.241s user 0.001s sys 0.000s
I20260806 20:35:43.529713 16971 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:43.531291 16996 log.cc:924] Time spent T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260806 20:35:43.532217 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.533068 14478 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:43.534184 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.497558 16820 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.534811 14469 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.521579 16696 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.541051 15150 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:43.542420 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:43.542842 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 198888us
W20260806 20:35:43.543394 15150 leader_election.cc:334] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.666s (SENT)
I20260806 20:35:43.545233 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.544762 14595 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:43.547758 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.548542 14478 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:43.549681 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.550179 14479 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:43.582463 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.547837 16980 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:43.545469 16935 raft_consensus.cc:686] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:43.582438 14547 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:43.549161 15181 raft_consensus.cc:2379] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 4: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:43.553161 16955 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:43.547199 16823 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.592000 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.566346 16905 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.545102 16855 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.592424 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.594699 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.613001 14630 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:43.613920 14547 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260806 20:35:43.614413 14547 pending_rounds.cc:76] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 5
I20260806 20:35:43.614684 14547 pending_rounds.cc:97] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 20:35:43.614956 14547 raft_consensus.cc:2866] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:43.615324 14547 pending_rounds.cc:97] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 20:35:43.615602 14547 raft_consensus.cc:2866] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:43.616822 14633 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:43.619149 14473 raft_consensus.cc:2379] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:43.568254 14595 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.553143 16870 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20260806 20:35:43.602578 16942 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.664618 14548 raft_consensus.cc:1226] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 20:35:43.667740 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.607501 16955 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:43.612615 16853 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.548427 16286 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.683362 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.667253 17001 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:43.643934 17002 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.660712 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.733531 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1641) took 1488 ms. Trace:
I20260806 20:35:43.734434 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1507) took 1448 ms. Trace:
I20260806 20:35:43.623625 16855 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.606793 14479 raft_consensus.cc:2396] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:43.691579 16915 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.744977 14473 raft_consensus.cc:2396] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:43.620759 16823 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.798158 14469 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
W20260806 20:35:43.743563 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 175365us
I20260806 20:35:43.799224 14595 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.800436 14595 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:43.764463 15150 leader_election.cc:334] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 1.539s (SENT)
I20260806 20:35:43.800916 15187 raft_consensus.cc:2354] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:43.752084 16955 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.766283 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:43.730667 16903 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.770236 14554 rpcz_store.cc:276] 0806 20:35:42.286433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.701082 (+414649us) service_pool.cc:227] Handling call
0806 20:35:42.701519 (+ 437us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:43.234969 (+533450us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:43.694634 (+459665us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b540013abd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:43.734375 (+ 39741us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1055466368}
I20260806 20:35:43.769418 14553 rpcz_store.cc:276] 0806 20:35:42.245333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.680337 (+435004us) service_pool.cc:227] Handling call
0806 20:35:42.680657 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:43.206308 (+525651us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:43.694307 (+487999us) spinlock_profiling.cc:243] Waited 503 ms on lock 0x7b54002db4d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:43.733447 (+ 39140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1157605632}
I20260806 20:35:43.742228 16624 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:43.802279 14595 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.802518 15150 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:43.762023 16854 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.806802 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.799403 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.809074 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.810227 14469 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:43.810240 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.813014 14546 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 2.
I20260806 20:35:43.813985 14626 raft_consensus.cc:2379] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:43.820079 17003 log.cc:924] Time spent T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.208s user 0.001s sys 0.000s
W20260806 20:35:43.822732 17007 log.cc:924] Time spent T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260806 20:35:43.768898 16853 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:43.827080 16990 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.613s user 0.000s sys 0.011s
I20260806 20:35:43.828419 16853 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:43.793951 17002 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:43.829869 16844 consensus_meta.cc:287] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.559s user 0.005s sys 0.005s
I20260806 20:35:43.830310 16844 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.830421 16965 raft_consensus.cc:3032] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:43.830796 16965 raft_consensus.cc:729] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:43.830940 17001 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.831012 16823 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.816504 14595 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.837401 14478 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:43.841243 14595 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.805424 16683 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.805760 17015 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:43.803203 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.762552 16935 consensus_queue.cc:237] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.857208 15181 raft_consensus.cc:2396] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 20:35:43.805675 16820 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.842509 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.844846 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:43.861783 14473 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.845167 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.831718 16947 raft_consensus.cc:3032] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:43.829897 16855 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.849151 16897 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:43.850250 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:43.851718 16972 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.000s sys 0.017s
I20260806 20:35:43.852947 17020 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:43.812319 16915 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:43.893356 16915 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:43.893891 16915 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:43.858878 14595 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.873066 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.863041 16714 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:43.830823 17002 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:43.875367 16965 consensus_queue.cc:260] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.902197 16965 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:43.863090 16947 raft_consensus.cc:729] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:43.903378 16947 consensus_queue.cc:260] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.904628 16947 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:43.907200 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 589702us
W20260806 20:35:43.876492 17005 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.276s user 0.001s sys 0.000s
I20260806 20:35:43.878492 16286 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.881415 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:43.881685 14470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 303) took 1805 ms (client timeout 1666 ms). Trace:
W20260806 20:35:43.915912 14470 rpcz_store.cc:269] 0806 20:35:42.076154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.088276 (+ 12122us) service_pool.cc:227] Handling call
0806 20:35:43.881459 (+1793183us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400117bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:43.881596 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1668103936}
I20260806 20:35:43.882197 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:43.883141 16946 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.884472 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:43.916821 14473 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:43.916941 14633 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:43.887342 14474 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:43.889381 14478 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:43.890789 14631 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 20:35:43.890453 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 399) took 1019 ms. Trace:
I20260806 20:35:43.917075 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.918430 15150 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:43.918802 14470 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:43.892688 15179 raft_consensus.cc:1226] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:43.897244 14627 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:43.809289 16903 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.899420 15185 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
W20260806 20:35:43.890365 16972 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s user 0.000s sys 0.017s
I20260806 20:35:43.912101 15151 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.933712 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:43.934389 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:43.913679 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 54893.1us
I20260806 20:35:43.936226 14478 raft_consensus.cc:2379] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:43.964637 15151 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.917913 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c"
dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:38502
W20260806 20:35:43.912806 17003 log.cc:924] Time spent T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260806 20:35:43.921063 16971 consensus_queue.cc:1046] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20260806 20:35:43.919566 16654 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.972285 14470 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:43.965492 16941 consensus_queue.cc:1046] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:35:43.920202 15183 rpcz_store.cc:276] 0806 20:35:42.871033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.872871 (+ 1838us) service_pool.cc:227] Handling call
0806 20:35:43.883650 (+1010779us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54006f04d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:43.883838 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":805954432}
I20260806 20:35:43.975648 16941 raft_consensus.cc:3032] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:43.915979 16714 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:43.975960 16654 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.976498 14550 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260806 20:35:43.976826 14550 pending_rounds.cc:76] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 5
I20260806 20:35:43.977034 14550 pending_rounds.cc:97] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260806 20:35:43.924540 16946 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.977262 14550 raft_consensus.cc:2866] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:43.978060 16654 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:43.978747 16946 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:43.975994 16941 raft_consensus.cc:729] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:43.979296 16844 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:43.979912 16941 consensus_queue.cc:260] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:43.981218 16941 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:43.978745 16987 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:43.980011 16972 consensus_queue.cc:1046] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:35:43.983950 16848 raft_consensus.cc:3032] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:43.927202 16286 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:43.984359 16848 raft_consensus.cc:729] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:43.984666 16286 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:43.985085 16286 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.985263 16848 consensus_queue.cc:260] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:43.986522 16848 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:43.986984 16834 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:43.964285 14633 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:43.989837 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:43.990481 14546 raft_consensus.cc:1261] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:43.990422 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:43.991240 14553 raft_consensus.cc:2379] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:43.992297 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:43.968055 14626 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:35:43.993367 14473 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 3.
I20260806 20:35:43.993561 14626 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:43.967990 14595 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.968822 14631 raft_consensus.cc:1226] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:43.923705 14475 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:43.925797 16875 raft_consensus.cc:3032] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:43.926729 16986 raft_consensus.cc:686] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:43.974375 15150 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:43.996322 14553 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:44.001888 15186 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:44.002077 14521 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:43.997131 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:43.973590 15151 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.997848 14595 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.925371 15181 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:43.915055 16943 consensus_queue.cc:1035] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260806 20:35:44.004233 14478 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:44.006525 15185 raft_consensus.cc:1261] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:44.008168 14552 raft_consensus.cc:1226] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []
I20260806 20:35:44.010233 14547 raft_consensus.cc:1226] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:35:44.012120 14552 raft_consensus.cc:1261] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:44.010211 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:44.014379 14553 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:44.015022 14546 raft_consensus.cc:1261] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:44.015926 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:44.022292 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.025823 14547 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:44.026656 16815 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:35:44.026778 15182 consensus_meta.cc:287] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.510s user 0.009s sys 0.002s
I20260806 20:35:44.027228 15182 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:35:44.027599 14595 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:44.027879 15151 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:43.996006 16875 raft_consensus.cc:729] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:44.028807 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:44.029366 16870 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:44.029237 16875 consensus_queue.cc:260] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.029960 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:44.030076 16870 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:35:44.030645 16875 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:44.031010 16946 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:35:44.031100 16853 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.031623 16946 raft_consensus.cc:686] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:44.031693 16853 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.032773 16884 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:44.032605 16946 consensus_queue.cc:237] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.033932 16853 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:44.031836 16870 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260806 20:35:44.028949 14625 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:44.025056 14633 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:35:44.037177 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.037531 16815 raft_consensus.cc:686] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:44.034462 16980 consensus_queue.cc:1046] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:44.038460 16815 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.001382 16956 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:44.040248 16956 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:44.040760 16956 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:43.983968 16972 raft_consensus.cc:3032] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:43.987515 16834 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.043704 14595 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:44.043802 14441 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:44.045245 14552 raft_consensus.cc:1226] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:44.047691 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:43.981855 16987 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.048143 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:43.996104 16873 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.054478 16897 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 20:35:44.085611 16966 log.cc:924] Time spent T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 1.056s user 0.008s sys 0.000s
I20260806 20:35:44.087404 14631 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:35:44.063175 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:44.093637 14444 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:44.115801 14444 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:44.115718 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:44.117038 14553 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:44.006211 17020 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:44.047641 17010 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:44.121212 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 78680.4us
I20260806 20:35:44.121593 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:44.114488 14626 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.128383 14548 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:35:44.081127 17012 log.cc:924] Time spent T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.272s user 0.000s sys 0.001s
I20260806 20:35:44.049880 16972 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:44.049113 16853 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.133894 14626 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:44.080777 16855 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:44.083225 17021 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:44.054279 17001 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:44.010102 16955 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.142155 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:44.057163 16951 consensus_queue.cc:1035] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20260806 20:35:44.144805 14442 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:44.131582 16974 consensus_meta.cc:287] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.709s user 0.011s sys 0.000s
I20260806 20:35:44.114425 16893 raft_consensus.cc:3032] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:44.113375 16941 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:35:44.150890 17028 log.cc:924] Time spent T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260806 20:35:44.151273 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.181018 14631 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:44.182672 14521 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:44.152542 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:44.184628 14555 raft_consensus.cc:3032] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:44.185036 14555 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:44.017772 16945 consensus_queue.cc:1035] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.115s
I20260806 20:35:44.141526 14518 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:44.123638 16705 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.017469 16969 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
I20260806 20:35:44.163522 14553 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:44.080906 16986 consensus_queue.cc:237] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.163518 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1704) took 1376 ms. Trace:
I20260806 20:35:44.134943 16714 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:44.173408 16903 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:44.097254 16873 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.172575 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.146402 16855 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:44.174940 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 131363us
I20260806 20:35:44.225306 14595 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:44.174937 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:44.119617 16753 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260806 20:35:44.131906 16820 consensus_queue.cc:1046] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260806 20:35:44.138616 16654 consensus_queue.cc:1046] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260806 20:35:44.171063 16884 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.206259 14553 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 20:35:44.082973 16987 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:44.269394 17004 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:44.175473 16930 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:44.146567 16853 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.176440 16974 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.174546 16893 raft_consensus.cc:729] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:44.154155 16872 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
W20260806 20:35:44.250765 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 75870.6us
I20260806 20:35:44.205479 16941 raft_consensus.cc:686] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:44.192162 14555 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.335247 14555 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:44.216991 16973 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.176872 16915 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.153285 17020 consensus_queue.cc:1035] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
I20260806 20:35:44.351532 14630 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:44.191828 16942 consensus_queue.cc:1046] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 20:35:44.274995 14547 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:35:44.276575 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 51365.5us
I20260806 20:35:44.211213 14477 rpcz_store.cc:276] 0806 20:35:42.786792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.786913 (+ 121us) service_pool.cc:227] Handling call
0806 20:35:42.800889 (+ 13976us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:42.875399 (+ 74510us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:42.875428 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:42.891617 (+ 16189us) log.cc:842] Serialized 26 byte log entry
0806 20:35:43.203109 (+311492us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:43.203151 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.153316 (+950165us) spinlock_profiling.cc:243] Waited 74.3 ms on lock 0x7b5400332cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:44.153365 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:44.153409 (+ 44us) raft_consensus.cc:1671] finished
0806 20:35:44.153420 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:44.163439 (+ 10019us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":170852736,"thread_start_us":258897,"threads_started":1,"wal-append.queue_time_us":372053}
I20260806 20:35:44.206979 16947 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260806 20:35:44.227198 16855 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:44.282936 17004 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:44.170938 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:44.380686 17004 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:44.381673 16955 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:44.359426 17027 log.cc:924] Time spent T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.270s user 0.000s sys 0.001s
I20260806 20:35:44.284610 16884 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.385052 16884 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:44.385515 16873 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:44.385679 16893 consensus_queue.cc:260] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.387331 16893 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:44.212340 16705 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.297226 14546 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:44.267436 16903 raft_consensus.cc:2741] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260806 20:35:44.305400 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 122828us
I20260806 20:35:44.272852 16905 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:44.310527 14632 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:44.395539 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1446) took 1058 ms. Trace:
I20260806 20:35:44.395869 14632 rpcz_store.cc:276] 0806 20:35:43.337367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.354604 (+ 17237us) service_pool.cc:227] Handling call
0806 20:35:43.358250 (+ 3646us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.310327 (+952077us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x7b54001c7b20. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b3483730 00007fb9b34f5300 00007fb9b34f6371 00007fb9b34f2edf 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc
0806 20:35:44.395265 (+ 84938us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.395481 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186783104}
W20260806 20:35:44.310911 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 169432us
I20260806 20:35:44.397126 14555 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:44.397641 14518 leader_election.cc:398] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4.
I20260806 20:35:44.397912 14518 leader_election.cc:401] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 20:35:44.269526 15178 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:44.192135 16972 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.309636 16962 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:44.360548 14595 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:44.326798 16987 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:44.273873 17025 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.234s
W20260806 20:35:44.404886 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 77660.9us
I20260806 20:35:44.379817 16974 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:44.358863 16996 log.cc:924] Time spent T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260806 20:35:44.333209 16930 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.385820 14477 raft_consensus.cc:1261] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:44.390172 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:44.423919 14518 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:44.340497 16624 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.000s
I20260806 20:35:44.398357 14596 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:44.424399 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1552) took 1092 ms. Trace:
I20260806 20:35:44.401674 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.425015 14477 rpcz_store.cc:276] 0806 20:35:43.331733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.359581 (+1027848us) service_pool.cc:227] Handling call
0806 20:35:44.385460 (+ 25879us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.424054 (+ 38594us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.424329 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:44.425806 14632 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:44.406859 14442 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:44.409204 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 78286.2us
I20260806 20:35:44.424268 14629 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:44.428141 14521 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:44.418907 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.431324 14625 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:44.433049 14521 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:44.408627 16873 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4.
I20260806 20:35:44.456243 14442 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:44.408082 16987 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:44.473101 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 49290.4us
I20260806 20:35:44.354890 16973 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.485230 14555 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 20:35:44.470662 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 110236us
I20260806 20:35:44.408066 15178 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 458) took 1073 ms. Trace:
I20260806 20:35:44.423118 16972 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:44.474427 14518 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:44.443593 16965 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.541s user 0.010s sys 0.001s
I20260806 20:35:44.453313 16654 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:44.501626 16654 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.448700 17002 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:44.509322 16955 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.510097 16955 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.457562 14477 raft_consensus.cc:1261] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:35:44.476967 14633 raft_consensus.cc:1261] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:44.489358 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:44.456400 16982 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260806 20:35:44.491384 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 456) took 1161 ms. Trace:
I20260806 20:35:44.442498 16524 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.518949 15187 rpcz_store.cc:276] 0806 20:35:43.329652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.453099 (+123447us) service_pool.cc:227] Handling call
0806 20:35:44.491207 (+1038108us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b540037f0d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:44.491331 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":685325568}
I20260806 20:35:44.519233 16524 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.522002 16524 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:44.466377 16995 log.cc:924] Time spent T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260806 20:35:44.450922 16624 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:44.389804 16999 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.312s
I20260806 20:35:44.454087 16820 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
I20260806 20:35:44.449651 16930 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.465724 16941 consensus_queue.cc:237] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.692607 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 338) took 1015 ms. Trace:
I20260806 20:35:44.484386 16987 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:44.667172 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 192817us
I20260806 20:35:44.721755 14518 leader_election.cc:398] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8.
I20260806 20:35:44.715035 14630 rpcz_store.cc:276] 0806 20:35:43.676948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.683006 (+ 6058us) service_pool.cc:227] Handling call
0806 20:35:44.692389 (+1009383us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b54002bedd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:44.692533 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1371942784}
I20260806 20:35:44.723573 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.603302 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:44.599589 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:44.738034 15184 raft_consensus.cc:2354] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:44.718256 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1554) took 1386 ms. Trace:
I20260806 20:35:44.739109 14477 rpcz_store.cc:276] 0806 20:35:43.332136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.426118 (+1093982us) service_pool.cc:227] Handling call
0806 20:35:44.457125 (+ 31007us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.717875 (+260750us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.718179 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:44.524551 14629 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:35:44.714493 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 221949us
I20260806 20:35:44.484323 16872 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:44.626012 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.465363 16971 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:44.738044 14518 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:44.581107 17035 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.370s user 0.001s sys 0.000s
W20260806 20:35:44.601025 17026 log.cc:924] Time spent T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.335s user 0.001s sys 0.001s
I20260806 20:35:44.746995 14633 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:35:44.622380 17037 log.cc:924] Time spent T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.289s user 0.001s sys 0.000s
I20260806 20:35:44.684798 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 572) took 1391 ms. Trace:
W20260806 20:35:44.600644 17029 log.cc:924] Time spent T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
W20260806 20:35:44.622758 17031 log.cc:924] Time spent T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.334s user 0.000s sys 0.001s
I20260806 20:35:44.747754 15184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 455) took 1419 ms. Trace:
W20260806 20:35:44.673244 17032 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.110s user 0.000s sys 0.006s
I20260806 20:35:44.599800 16873 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.749749 16971 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:44.592684 16962 raft_consensus.cc:2741] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:35:44.756803 14444 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:44.677421 16286 consensus_meta.cc:287] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.692s user 0.010s sys 0.002s
W20260806 20:35:44.766784 16875 consensus_meta.cc:287] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.736s user 0.010s sys 0.005s
W20260806 20:35:44.728183 16848 consensus_meta.cc:287] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.741s user 0.007s sys 0.006s
I20260806 20:35:44.760632 14633 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:44.602463 16910 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:35:44.784642 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 328461us
I20260806 20:35:44.591174 17002 raft_consensus.cc:686] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:44.611675 16973 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:44.763046 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 244309us
I20260806 20:35:44.765550 16945 raft_consensus.cc:3032] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:44.466768 16974 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:44.763804 14477 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:44.749815 17015 consensus_meta.cc:287] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.881s user 0.012s sys 0.000s
I20260806 20:35:44.747535 16872 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:44.612340 16705 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:44.504352 16796 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:44.768857 15184 rpcz_store.cc:276] 0806 20:35:43.328266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.387215 (+ 58949us) service_pool.cc:227] Handling call
0806 20:35:44.599382 (+1212167us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:44.747696 (+148314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6658304}
I20260806 20:35:44.593101 16818 consensus_queue.cc:1046] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.174s
I20260806 20:35:44.491593 15178 rpcz_store.cc:276] 0806 20:35:43.330610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.551940 (+221330us) service_pool.cc:227] Handling call
0806 20:35:44.403950 (+852010us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:44.624619 16853 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:44.781685 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 43755.8us
I20260806 20:35:44.814491 14518 leader_election.cc:398] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
W20260806 20:35:44.782186 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 292902us
I20260806 20:35:44.757954 17021 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:44.687668 16937 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:44.773751 16955 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:44.750015 16855 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.771976 16856 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:44.803300 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1564) took 1463 ms. Trace:
I20260806 20:35:44.885424 14477 rpcz_store.cc:276] 0806 20:35:43.339643 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.740479 (+1400836us) service_pool.cc:227] Handling call
0806 20:35:44.761557 (+ 21078us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.803008 (+ 41451us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.803233 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:44.804147 14442 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:44.751890 14550 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.002s sys 0.008s
W20260806 20:35:44.886647 14550 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.766s user 0.002s sys 0.009s
I20260806 20:35:44.808327 17004 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.776753 16971 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:44.783275 16286 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.786612 16910 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:44.822100 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:44.762334 15179 rpcz_store.cc:276] 0806 20:35:43.293232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.295974 (+ 2742us) service_pool.cc:227] Handling call
0806 20:35:43.890367 (+594393us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:43.934494 (+ 44127us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.684507 (+750013us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b54006f04d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:44.684707 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1344037248}
I20260806 20:35:44.789505 16945 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:44.825251 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:44.838644 14478 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.008s sys 0.014s
W20260806 20:35:44.904320 14478 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.759s user 0.009s sys 0.014s
I20260806 20:35:44.902074 14477 raft_consensus.cc:1261] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:44.789280 16942 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:44.810281 16935 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.004s sys 0.046s
I20260806 20:35:44.804713 16654 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:44.869808 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:44.916363 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:44.809933 17015 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:44.810546 16872 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:44.875171 15151 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:44.780370 16873 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:44.871449 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:44.887039 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 82979us
I20260806 20:35:44.812414 16974 raft_consensus.cc:686] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:44.768620 16870 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260806 20:35:44.799108 16962 consensus_queue.cc:1046] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260806 20:35:44.868155 16818 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:44.905345 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1567) took 1564 ms. Trace:
W20260806 20:35:44.908636 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 151933us
I20260806 20:35:44.916507 14628 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:44.917174 14548 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:44.810142 16796 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:44.919768 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 94666.4us
W20260806 20:35:44.928737 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 106709us
I20260806 20:35:44.926236 14627 raft_consensus.cc:1226] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:44.904214 16910 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:44.923671 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48611.1us
W20260806 20:35:44.917230 16935 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s user 0.005s sys 0.046s
I20260806 20:35:44.871765 16937 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:44.901430 17046 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:44.966120 14546 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:44.921391 16930 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.006469 14521 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:44.901532 17061 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:44.926481 17072 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:45.029325 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:45.029944 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:44.915648 16986 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:44.868355 17021 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.003928 14477 rpcz_store.cc:276] 0806 20:35:43.340745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.886323 (+1545578us) service_pool.cc:227] Handling call
0806 20:35:44.901657 (+ 15334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.905023 (+ 3366us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.905278 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:44.993621 16856 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.962623 16844 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:44.924693 17001 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:44.998699 16852 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:45.028383 14518 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:45.045974 15151 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:35:45.046643 16962 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:44.924284 17040 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:44.924721 16942 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.047041 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:45.245748 14548 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:45.032992 16705 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.041765 16524 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:45.005132 16818 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:45.000272 16947 raft_consensus.cc:3032] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:35:45.059302 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:45.021998 16796 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:45.064491 17069 log.cc:924] Time spent T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260806 20:35:45.027189 16999 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.289400 16999 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.290825 16873 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.292045 16999 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.032044 16897 consensus_queue.cc:1035] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260806 20:35:45.072562 16855 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.281627 16947 raft_consensus.cc:729] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:45.294353 17063 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.292s user 0.002s sys 0.000s
I20260806 20:35:45.074245 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:45.043807 17049 log.cc:924] Time spent T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.419s user 0.002s sys 0.000s
I20260806 20:35:45.032259 16683 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20260806 20:35:45.080473 15187 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:45.041430 16848 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.308084 16848 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:45.309841 17071 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
W20260806 20:35:45.310277 14551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 337) took 2004 ms (client timeout 1967 ms). Trace:
W20260806 20:35:45.310719 14551 rpcz_store.cc:269] 0806 20:35:43.305431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.519271 (+213840us) service_pool.cc:227] Handling call
0806 20:35:45.310023 (+1790752us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54003e94d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.310201 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2939315584}
I20260806 20:35:45.070921 16286 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.057822 17047 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:45.174671 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:45.033227 16955 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260806 20:35:45.095476 17046 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.095293 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:44.922897 17002 consensus_queue.cc:237] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.105875 16822 consensus_queue.cc:1035] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
W20260806 20:35:45.230286 16815 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.011s sys 0.011s
I20260806 20:35:45.145641 16856 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.228675 16844 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:45.227547 16915 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.236534 16962 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:45.281469 16705 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.284621 17030 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:35:45.690444 16815 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.108s user 0.017s sys 0.032s
I20260806 20:35:45.250730 17040 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.281675 16818 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:45.639060 17065 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:45.293797 17010 consensus_meta.cc:287] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.154s user 0.004s sys 0.009s
I20260806 20:35:45.237617 17015 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:45.182756 16974 consensus_queue.cc:237] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.245607 17001 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:45.315753 17058 log.cc:924] Time spent T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.415s user 0.001s sys 0.001s
W20260806 20:35:45.646665 16972 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.146s user 0.009s sys 0.006s
W20260806 20:35:45.312778 17076 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.281s user 0.000s sys 0.001s
W20260806 20:35:45.530413 17077 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.103s user 0.000s sys 0.002s
W20260806 20:35:45.351651 17069 log.cc:924] Time spent T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
W20260806 20:35:45.314584 17060 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
W20260806 20:35:45.427345 17066 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.413s user 0.001s sys 0.000s
W20260806 20:35:45.436553 16987 consensus_meta.cc:287] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.692s user 0.009s sys 0.005s
I20260806 20:35:45.426339 17075 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:45.435719 16956 consensus_meta.cc:287] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.394s user 0.012s sys 0.003s
I20260806 20:35:45.694635 17052 raft_consensus.cc:3032] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:45.225160 16980 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.755679 16980 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.756664 16956 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.952975 16935 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:45.841313 14444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.836s user 0.015s sys 0.033s
I20260806 20:35:45.205602 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:45.959080 14476 raft_consensus.cc:2379] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 8: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:45.959740 14476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1557) took 2622 ms (client timeout 1989 ms). Trace:
I20260806 20:35:45.172549 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:45.961349 14477 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:45.843123 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1719) took 2517 ms. Trace:
I20260806 20:35:45.754616 16987 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.964974 16820 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.963871 14474 rpcz_store.cc:276] 0806 20:35:43.325455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.273806 (+948351us) service_pool.cc:227] Handling call
0806 20:35:44.274210 (+ 404us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.276853 (+ 2643us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.276898 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.332829 (+ 55931us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.790846 (+458017us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:44.790895 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.791129 (+ 234us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.842173 (+1051044us) raft_consensus.cc:1671] finished
0806 20:35:45.842257 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.842788 (+ 531us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1859,"raft.queue_time_us":2153,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":4672,"thread_start_us":386192,"threads_started":1,"wal-append.queue_time_us":925491}
I20260806 20:35:45.965914 16820 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.967129 16987 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:45.230176 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 201881us
W20260806 20:35:45.851526 14478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.946s user 0.004s sys 0.004s
I20260806 20:35:45.296666 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:46.085654 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1532) took 1111 ms. Trace:
I20260806 20:35:46.086081 14548 rpcz_store.cc:276] 0806 20:35:44.974314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.246575 (+272261us) service_pool.cc:227] Handling call
0806 20:35:46.085473 (+838898us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:46.087285 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:46.079648 16987 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:45.853257 14546 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.816s user 0.026s sys 0.012s
I20260806 20:35:46.088547 14546 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:46.088591 16987 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.854301 14551 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:46.090245 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 351) took 1203 ms. Trace:
I20260806 20:35:46.090626 14551 rpcz_store.cc:276] 0806 20:35:44.886351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.311733 (+425382us) service_pool.cc:227] Handling call
0806 20:35:45.853234 (+541501us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:46.089906 (+236672us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:46.090163 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:46.090339 14369 catalog_manager.cc:5495] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 5, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 5 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:46.091470 16987 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.350014 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1525) took 1307 ms. Trace:
I20260806 20:35:46.092653 14547 rpcz_store.cc:276] 0806 20:35:44.042430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.042620 (+ 190us) service_pool.cc:227] Handling call
0806 20:35:45.349760 (+1307140us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540018ecd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.349931 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":495399168}
I20260806 20:35:46.092347 14369 catalog_manager.cc:5495] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:45.853590 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1655) took 1864 ms. Trace:
I20260806 20:35:45.855581 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 347) took 1715 ms. Trace:
I20260806 20:35:45.856343 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1666) took 1113 ms. Trace:
I20260806 20:35:45.867616 14549 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:35:45.874756 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1799) took 1478 ms. Trace:
I20260806 20:35:45.873548 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 344) took 1431 ms. Trace:
I20260806 20:35:45.858021 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 344) took 1718 ms. Trace:
W20260806 20:35:45.875321 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1546) took 2549 ms (client timeout 1819 ms). Trace:
I20260806 20:35:45.875836 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1808) took 1056 ms. Trace:
I20260806 20:35:45.513857 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1791) took 1492 ms. Trace:
I20260806 20:35:45.877180 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1460) took 1203 ms. Trace:
I20260806 20:35:45.877029 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1798) took 1480 ms. Trace:
W20260806 20:35:45.877857 14633 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.002s sys 0.005s
W20260806 20:35:45.879544 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 700905us
I20260806 20:35:45.850673 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1711) took 3058 ms. Trace:
W20260806 20:35:45.453949 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 408033us
I20260806 20:35:45.073220 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:45.313699 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 571) took 2134 ms. Trace:
I20260806 20:35:45.909759 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 463) took 2572 ms. Trace:
I20260806 20:35:45.909659 15182 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
W20260806 20:35:46.113109 14633 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.003s sys 0.005s
I20260806 20:35:46.112974 15185 rpcz_store.cc:276] 0806 20:35:43.337367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.929803 (+592436us) service_pool.cc:227] Handling call
0806 20:35:44.003220 (+ 73417us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.007003 (+ 3783us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:45.909190 (+1902187us) spinlock_profiling.cc:243] Waited 73.1 ms on lock 0x7b54003be2d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.909507 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":168087424}
I20260806 20:35:45.913987 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56934 (request call id 1) took 2382 ms. Trace:
I20260806 20:35:46.115075 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 469) took 1085 ms. Trace:
I20260806 20:35:46.114967 15186 rpcz_store.cc:276] 0806 20:35:43.531029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.002568 (+471539us) service_pool.cc:227] Handling call
0806 20:35:45.911416 (+1908848us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b54007003d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b8438c6e 00007fb9b1b2f807 00007fb9b1b2f792 00007fb9b1b2f6c1 00007fb9b1b2f63c 00007fb9b1b2e8f2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.913663 (+ 2247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71362816}
I20260806 20:35:46.115660 15182 rpcz_store.cc:276] 0806 20:35:45.030114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.033018 (+ 2904us) service_pool.cc:227] Handling call
0806 20:35:46.114760 (+1081742us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54003d0ed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:46.114965 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":502137088}
I20260806 20:35:45.284919 16524 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.113040 15180 rpcz_store.cc:276] 0806 20:35:43.179568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.183087 (+ 3519us) service_pool.cc:227] Handling call
0806 20:35:43.236213 (+ 53126us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:43.296584 (+ 60371us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:45.285947 (+1989363us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b54004510d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.313580 (+ 27633us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87985408}
I20260806 20:35:46.100538 14471 rpcz_store.cc:276] 0806 20:35:42.791935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.806480 (+ 14545us) service_pool.cc:227] Handling call
0806 20:35:42.806699 (+ 219us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:43.467051 (+660352us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b1800272d40. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b0598f96 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34870d8 00007fb9b34f2fed 00007fb9b34f299e
0806 20:35:45.041016 (+1573965us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b54000b07a0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34870d8 00007fb9b34f2fed 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c
0806 20:35:45.041078 (+ 62us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:45.041099 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:45.050484 (+ 9385us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:45.050578 (+ 94us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:45.850120 (+799542us) spinlock_profiling.cc:243] Waited 68.6 ms on lock 0x7b54003c4e54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:45.850187 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.850447 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14831,"raft.queue_time_us":11242,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":10082,"spinlock_wait_cycles":178484352}
I20260806 20:35:45.927780 16696 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.929057 16815 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.929575 16873 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:45.647982 16910 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.606s user 0.005s sys 0.009s
I20260806 20:35:45.930753 16951 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:45.705250 16962 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:45.931630 16705 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.697693 16844 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:46.133536 16951 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:46.112375 15151 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.967s (SENT) [suppressed 32 similar messages]
I20260806 20:35:46.123961 16815 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.139928 16815 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.311601 16848 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.724267 16856 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:45.647981 16872 consensus_meta.cc:287] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.640s user 0.005s sys 0.005s
I20260806 20:35:46.140844 16873 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.141850 16872 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.143420 16873 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:46.144114 16872 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.934120 16834 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.934489 16955 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.145663 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1536) took 1170 ms. Trace:
I20260806 20:35:46.145934 16955 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.146134 14547 rpcz_store.cc:276] 0806 20:35:44.975436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.093668 (+1118232us) service_pool.cc:227] Handling call
0806 20:35:46.094099 (+ 431us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:46.144486 (+ 50387us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:46.144519 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:46.144590 (+ 71us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:46.144641 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:46.145400 (+ 759us) spinlock_profiling.cc:243] Waited 49.9 ms on lock 0x7b54001fc2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:46.145445 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:46.145579 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":500,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":152,"spinlock_wait_cycles":114835200}
I20260806 20:35:45.934984 16852 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:46.148177 16852 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.149848 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1720) took 2824 ms. Trace:
I20260806 20:35:46.150543 16852 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.550225 16954 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:46.150249 14470 rpcz_store.cc:276] 0806 20:35:43.325751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.292880 (+967129us) service_pool.cc:227] Handling call
0806 20:35:45.846927 (+1554047us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:45.946866 (+ 99939us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:45.946914 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:45.947817 (+ 903us) log.cc:842] Serialized 26 byte log entry
0806 20:35:46.078387 (+130570us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:46.078443 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:46.080861 (+ 2418us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:46.148522 (+ 67661us) raft_consensus.cc:1671] finished
0806 20:35:46.148539 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:46.149450 (+ 911us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54002323d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:46.149688 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98919,"raft.queue_time_us":102298,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":307,"spinlock_wait_cycles":3572692224,"thread_start_us":130015,"threads_started":1,"wal-append.queue_time_us":196621}
I20260806 20:35:46.151731 16954 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.152824 16910 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.154250 16954 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:45.428966 16947 consensus_queue.cc:260] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.155511 16910 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:46.156160 16947 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:46.156781 17002 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:46.157455 17002 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.159060 16818 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.348083 17004 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.739848 17001 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.162581 17001 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:45.736428 17015 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.486083 17021 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.715796 17030 raft_consensus.cc:686] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:46.098527 14626 rpcz_store.cc:276] 0806 20:35:44.396343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.396485 (+ 142us) service_pool.cc:227] Handling call
0806 20:35:44.396808 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.400804 (+ 3996us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.400834 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.408800 (+ 7966us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.511523 (+102723us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:44.511581 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.511675 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.874225 (+1362550us) raft_consensus.cc:1671] finished
0806 20:35:45.874259 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.874592 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22457,"raft.queue_time_us":7007,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":17211,"thread_start_us":37755,"threads_started":1,"wal-append.queue_time_us":315124}
I20260806 20:35:46.165767 17030 consensus_queue.cc:237] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.166698 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:46.166707 16823 consensus_queue.cc:1159] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "148bf2a11dba423ab364371f0458e871" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:35:45.764820 17040 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:45.753890 17052 raft_consensus.cc:729] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:45.703668 17047 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:45.784055 17056 log.cc:924] Time spent T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 1.160s user 0.000s sys 0.004s
I20260806 20:35:45.616501 17059 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.170773 17052 consensus_queue.cc:260] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.171119 17059 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.165758 17015 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:46.172389 17052 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:46.172976 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:45.728643 17065 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:45.784348 17074 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.719s user 0.000s sys 0.003s
I20260806 20:35:46.180545 16852 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:45.754328 17075 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:46.181394 16852 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.182221 17075 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:45.945591 16986 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:45.948999 16980 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.949301 16854 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.703375 16945 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.950688 16903 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.191632 16986 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.191485 16854 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.193259 16945 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:45.840402 14442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.865s user 0.008s sys 0.042s
W20260806 20:35:46.194209 14442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.219s user 0.009s sys 0.042s
I20260806 20:35:46.195441 17004 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:46.195989 16853 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:45.954821 14444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.949s user 0.015s sys 0.033s
I20260806 20:35:46.196270 17004 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.196877 16853 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.198623 16834 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.199409 17004 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:46.199221 16834 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.194901 16854 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:46.201354 16854 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.202194 17047 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:46.200204 17004 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.202831 17047 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.203227 17004 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.203570 17047 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.967041 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:45.844234 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1563) took 2504 ms (client timeout 2374 ms). Trace:
W20260806 20:35:45.846248 14473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1540) took 2522 ms (client timeout 2831 ms). Trace:
I20260806 20:35:46.014400 16820 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:45.996747 16935 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.040305 16999 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.003289 16946 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.001302 16969 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.228844 16946 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.229113 16969 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.231130 16946 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:46.011668 16908 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.304s user 0.010s sys 0.010s
W20260806 20:35:46.231740 16908 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.534s user 0.014s sys 0.014s
I20260806 20:35:46.042217 16937 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:46.004618 16956 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:45.962332 16796 consensus_meta.cc:287] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.673s user 0.005s sys 0.010s
W20260806 20:35:46.011089 16971 consensus_meta.cc:287] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.105s user 0.017s sys 0.000s
W20260806 20:35:45.960294 14476 rpcz_store.cc:269] 0806 20:35:43.337236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.413072 (+ 75836us) service_pool.cc:227] Handling call
0806 20:35:45.205418 (+1792346us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:45.959603 (+754185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49348864}
I20260806 20:35:46.233035 16796 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:45.963531 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:45.949102 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1797) took 1553 ms. Trace:
I20260806 20:35:46.039489 16942 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:46.087879 17097 log.cc:924] Time spent T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.069s user 0.003s sys 0.000s
I20260806 20:35:46.089354 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 349) took 1218 ms. Trace:
W20260806 20:35:45.877369 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1558) took 2539 ms (client timeout 1652 ms). Trace:
I20260806 20:35:45.402833 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:46.101354 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1531) took 1127 ms. Trace:
I20260806 20:35:46.096113 14553 rpcz_store.cc:276] 0806 20:35:44.140044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.278808 (+138764us) service_pool.cc:227] Handling call
0806 20:35:44.279162 (+ 354us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:44.285253 (+ 6091us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.285301 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:44.303986 (+ 18685us) log.cc:842] Serialized 52 byte log entry
0806 20:35:44.427761 (+123775us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:44.504914 (+ 77153us) log.cc:842] Serialized 14 byte log entry
0806 20:35:44.505146 (+ 232us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.505233 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.307449 (+802216us) raft_consensus.cc:1671] finished
0806 20:35:45.307482 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.565971 (+258489us) spinlock_profiling.cc:243] Waited 7.39 ms on lock 0x7b18003aa680. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:45.855300 (+289329us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48914,"prepare.queue_time_us":154498,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":43,"raft.queue_time_us":80453,"raft.run_cpu_time_us":844,"raft.run_wall_time_us":116007,"spinlock_wait_cycles":16997632,"thread_start_us":75783,"threads_started":1,"wal-append.queue_time_us":279943}
W20260806 20:35:46.102610 14478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.197s user 0.005s sys 0.005s
I20260806 20:35:46.102094 14595 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:46.099145 14629 rpcz_store.cc:276] 0806 20:35:44.673796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.741216 (+ 67420us) service_pool.cc:227] Handling call
0806 20:35:44.741610 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.741867 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.741902 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.751627 (+ 9725us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.944951 (+193324us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:44.945008 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.945121 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.876492 (+931371us) raft_consensus.cc:1671] finished
0806 20:35:45.876556 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.876938 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":175064,"threads_started":1,"wal-append.queue_time_us":512011}
I20260806 20:35:46.098290 14555 rpcz_store.cc:276] 0806 20:35:44.743313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.743523 (+ 210us) service_pool.cc:227] Handling call
0806 20:35:44.744847 (+ 1324us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.783486 (+ 38639us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.783517 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:44.783553 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:44.783594 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:45.855753 (+1072159us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b54003e85d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:45.855835 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.856138 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88417664}
W20260806 20:35:46.098806 14479 rpcz_store.cc:269] 0806 20:35:43.325370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.401763 (+ 76393us) service_pool.cc:227] Handling call
0806 20:35:45.874811 (+2473048us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540030f054. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.875062 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331296000}
I20260806 20:35:46.095297 14550 rpcz_store.cc:276] 0806 20:35:43.988842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.988982 (+ 140us) service_pool.cc:227] Handling call
0806 20:35:44.023646 (+ 34664us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.120184 (+ 96538us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.120216 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.120679 (+ 463us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.887106 (+766427us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:44.887148 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.887225 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.852743 (+965518us) raft_consensus.cc:1671] finished
0806 20:35:45.852794 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.853289 (+ 495us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99390,"raft.queue_time_us":97005,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":10804,"thread_start_us":766097,"threads_started":1,"wal-append.queue_time_us":696700}
I20260806 20:35:46.098589 14625 rpcz_store.cc:276] 0806 20:35:44.441891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.442097 (+ 206us) service_pool.cc:227] Handling call
0806 20:35:44.472877 (+ 30780us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.473232 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.473262 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.473683 (+ 421us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.815885 (+342202us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:44.815910 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.815986 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.872844 (+1056858us) raft_consensus.cc:1671] finished
0806 20:35:45.872885 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.873300 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33920,"raft.queue_time_us":12126,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":6890,"thread_start_us":221541,"threads_started":1,"wal-append.queue_time_us":537585}
I20260806 20:35:46.108695 14627 rpcz_store.cc:276] 0806 20:35:44.021416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.022687 (+ 1271us) service_pool.cc:227] Handling call
0806 20:35:44.032437 (+ 9750us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:45.018116 (+985679us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:45.513481 (+495365us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b540006f4d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:45.513766 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2054900096}
I20260806 20:35:45.924903 16286 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:45.926875 16654 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.122751 16696 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.123071 16974 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:46.122974 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:46.123983 16524 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:46.159659 17002 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:45.734118 17010 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:46.167608 14626 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:45.703828 17046 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.174800 14548 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:46.177100 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:46.177755 14547 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:46.136227 16951 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:46.191895 16903 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:45.953123 15181 raft_consensus.cc:1226] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: []
I20260806 20:35:46.201875 16854 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:45.842211 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 744947us
I20260806 20:35:45.965118 16855 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:46.098982 14628 rpcz_store.cc:276] 0806 20:35:44.819720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.006838 (+187118us) service_pool.cc:227] Handling call
0806 20:35:45.007245 (+ 407us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:45.025440 (+ 18195us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:45.025487 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:45.046226 (+ 20739us) log.cc:842] Serialized 26 byte log entry
0806 20:35:45.237792 (+191566us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:45.237876 (+ 84us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:45.238085 (+ 209us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.875326 (+637241us) raft_consensus.cc:1671] finished
0806 20:35:45.875359 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.875631 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7823,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":5178,"thread_start_us":90825,"threads_started":1,"wal-append.queue_time_us":676193}
I20260806 20:35:46.190662 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 346) took 1403 ms. Trace:
I20260806 20:35:46.227172 14474 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:46.176191 17059 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:46.100086 14631 rpcz_store.cc:276] 0806 20:35:44.396010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.402066 (+ 6056us) service_pool.cc:227] Handling call
0806 20:35:44.440400 (+ 38334us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.491019 (+ 50619us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.491053 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.491535 (+ 482us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.762768 (+271233us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:44.784519 (+ 21751us) log.cc:842] Serialized 14 byte log entry
0806 20:35:44.784793 (+ 274us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.784868 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.567734 (+782866us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b18000fa7e0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:45.738832 (+171098us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x7b18000fa7e0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb764 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:45.876472 (+137640us) raft_consensus.cc:1671] finished
0806 20:35:45.876518 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.876808 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61947,"prepare.queue_time_us":21600,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":64083,"raft.run_cpu_time_us":406,"raft.run_wall_time_us":47253,"spinlock_wait_cycles":18335744,"thread_start_us":159621,"threads_started":1,"wal-append.queue_time_us":288977}
W20260806 20:35:46.204664 14469 rpcz_store.cc:269] 0806 20:35:43.339245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.799039 (+459794us) service_pool.cc:227] Handling call
0806 20:35:45.843665 (+2044626us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5400116cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.843943 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2331904}
I20260806 20:35:46.098721 14554 rpcz_store.cc:276] 0806 20:35:44.139179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.140547 (+ 1368us) service_pool.cc:227] Handling call
0806 20:35:44.161229 (+ 20682us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.409012 (+247783us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.409079 (+ 67us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.409460 (+ 381us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.476539 (+ 67079us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:44.476579 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:45.857023 (+1380444us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54002d9bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:45.857122 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.857235 (+ 113us) raft_consensus.cc:1671] finished
0806 20:35:45.857260 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.857789 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37710,"raft.queue_time_us":430,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":23051,"spinlock_wait_cycles":568951552,"thread_start_us":66768,"threads_started":1,"wal-append.queue_time_us":189293,"wal-append.run_cpu_time_us":2966,"wal-append.run_wall_time_us":1013274}
W20260806 20:35:47.002254 15151 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.008s sys 0.026s
W20260806 20:35:47.260224 15151 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.125s user 0.014s sys 0.030s
W20260806 20:35:47.260813 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 1.14854e+06us
W20260806 20:35:46.204732 14473 rpcz_store.cc:269] 0806 20:35:43.323866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.917499 (+593633us) service_pool.cc:227] Handling call
0806 20:35:45.845798 (+1928299us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b54002323d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.846030 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":134194048}
I20260806 20:35:46.233119 16955 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:47.262631 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 177001us
W20260806 20:35:47.263453 14473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1571) took 2068 ms (client timeout 1959 ms). Trace:
W20260806 20:35:47.263693 14631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1812) took 1911 ms (client timeout 1753 ms). Trace:
I20260806 20:35:47.231170 14549 rpcz_store.cc:276] 0806 20:35:44.973976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.866379 (+892403us) service_pool.cc:227] Handling call
0806 20:35:45.866936 (+ 557us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:46.098405 (+231469us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:46.101244 (+ 2839us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.264215 14631 rpcz_store.cc:269] 0806 20:35:45.352361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.263404 (+1911043us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:47.263603 (+ 199us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:35:46.184718 16818 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:47.265913 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1811) took 1937 ms (client timeout 1657 ms). Trace:
W20260806 20:35:47.270308 14626 rpcz_store.cc:269] 0806 20:35:45.294663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.166242 (+871579us) service_pool.cc:227] Handling call
0806 20:35:47.231939 (+1065697us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:46.234030 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:47.232468 14546 rpcz_store.cc:276] 0806 20:35:44.870674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.870885 (+ 211us) service_pool.cc:227] Handling call
0806 20:35:46.089261 (+1218376us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:46.680497 16820 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:47.231676 14475 rpcz_store.cc:269] 0806 20:35:43.337588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:43.361633 (+ 24045us) service_pool.cc:227] Handling call
0806 20:35:45.876982 (+2515349us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b54001a5ed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:45.877186 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":949464448}
I20260806 20:35:47.278223 14476 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:47.278678 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:47.279680 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 328) took 1076 ms. Trace:
I20260806 20:35:46.095940 14369 catalog_manager.cc:5495] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 2 to 4, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 148bf2a11dba423ab364371f0458e871 (127.12.248.132). New cstate: current_term: 4 leader_uuid: "148bf2a11dba423ab364371f0458e871" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:47.268707 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1676) took 1102 ms. Trace:
I20260806 20:35:46.322326 16875 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:47.272946 16286 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.274479 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:47.274987 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:47.274485 16796 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.275810 14518 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:47.276989 14632 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:47.277245 17002 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:47.069684 16844 consensus_meta.cc:287] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.936s user 0.006s sys 0.005s
W20260806 20:35:47.067589 16962 consensus_meta.cc:287] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.939s user 0.008s sys 0.004s
W20260806 20:35:47.263927 14473 rpcz_store.cc:269] 0806 20:35:45.196028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.263179 (+2067151us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:47.263362 (+ 183us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 20:35:47.305476 14477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1728) took 2227 ms (client timeout 1612 ms). Trace:
I20260806 20:35:47.408869 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.282361 14547 rpcz_store.cc:276] 0806 20:35:46.166438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.172682 (+ 6244us) service_pool.cc:227] Handling call
0806 20:35:47.268657 (+1095975us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.409933 14627 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:47.410429 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1819) took 1182 ms. Trace:
I20260806 20:35:47.410756 14627 rpcz_store.cc:276] 0806 20:35:46.227879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.334303 (+1106424us) service_pool.cc:227] Handling call
0806 20:35:47.410345 (+ 76042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.410971 14547 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260806 20:35:47.411901 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 180674us
I20260806 20:35:47.412825 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1671) took 1521 ms. Trace:
I20260806 20:35:47.231246 14633 rpcz_store.cc:276] 0806 20:35:44.787511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.788133 (+ 622us) service_pool.cc:227] Handling call
0806 20:35:44.788673 (+ 540us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:45.174038 (+385365us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:45.352194 (+178156us) log.cc:842] Serialized 14 byte log entry
0806 20:35:46.113957 (+761763us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:46.114062 (+ 105us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:46.114177 (+ 115us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:46.127284 (+ 13107us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b540020e254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:46.127351 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:46.190597 (+ 63246us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":549009,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":2528,"raft.queue_time_us":773,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":261,"spinlock_wait_cycles":885698304,"thread_start_us":918962,"threads_started":2,"wal-append.queue_time_us":398470,"wal-append.run_cpu_time_us":2200,"wal-append.run_wall_time_us":1270972}
I20260806 20:35:47.232761 14624 rpcz_store.cc:276] 0806 20:35:44.395660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.395795 (+ 135us) service_pool.cc:227] Handling call
0806 20:35:44.399613 (+ 3818us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:44.401318 (+ 1705us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:44.401355 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:44.401775 (+ 420us) log.cc:842] Serialized 26 byte log entry
0806 20:35:44.757215 (+355440us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:44.757275 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:44.757373 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:45.948632 (+1191259us) raft_consensus.cc:1671] finished
0806 20:35:45.948679 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:45.948912 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":169846,"raft.queue_time_us":23485,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":12492,"thread_start_us":260309,"threads_started":1,"wal-append.queue_time_us":535340,"wal-append.run_cpu_time_us":38616,"wal-append.run_wall_time_us":1986146}
I20260806 20:35:47.413779 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.413982 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:47.414695 14624 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:35:47.415156 14547 rpcz_store.cc:276] 0806 20:35:45.891493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.410135 (+1518642us) service_pool.cc:227] Handling call
0806 20:35:47.410535 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.412510 (+ 1975us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.412739 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.416772 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:47.418056 16696 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.419512 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1673) took 1527 ms. Trace:
I20260806 20:35:47.420094 14548 rpcz_store.cc:276] 0806 20:35:45.892223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.086903 (+194680us) service_pool.cc:227] Handling call
0806 20:35:47.419314 (+1332411us) spinlock_profiling.cc:243] Waited 86.8 ms on lock 0x7b5400355f54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:47.419439 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199553920}
W20260806 20:35:47.420938 17075 consensus_meta.cc:287] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.238s user 0.009s sys 0.005s
I20260806 20:35:47.421056 16852 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.421381 17075 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:47.064334 16945 consensus_meta.cc:287] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.871s user 0.009s sys 0.000s
I20260806 20:35:47.414624 14633 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:46.225193 16910 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.354537 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1669) took 1464 ms. Trace:
W20260806 20:35:47.396384 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1723) took 2315 ms (client timeout 1882 ms). Trace:
W20260806 20:35:47.335314 14470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1727) took 2282 ms (client timeout 1815 ms). Trace:
I20260806 20:35:47.353827 14630 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:47.280197 17052 consensus_meta.cc:287] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.107s user 0.005s sys 0.004s
I20260806 20:35:47.408247 14369 catalog_manager.cc:5495] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:47.456416 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1672) took 1541 ms. Trace:
I20260806 20:35:47.463436 14369 catalog_manager.cc:5495] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:47.463178 14547 rpcz_store.cc:276] 0806 20:35:45.891860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.416176 (+1524316us) service_pool.cc:227] Handling call
0806 20:35:47.416616 (+ 440us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.416968 (+ 352us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.417013 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:47.432564 (+ 15551us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:47.432623 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.432701 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:47.432856 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.354180 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.032618 17021 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:47.423077 17065 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.423378 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.423935 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:47.436012 14479 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:47.436829 14470 rpcz_store.cc:269] 0806 20:35:45.052637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.152230 (+1099593us) service_pool.cc:227] Handling call
0806 20:35:47.335216 (+1182986us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.437297 14469 rpcz_store.cc:269] 0806 20:35:45.019486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.270113 (+2250627us) service_pool.cc:214] Skipping call since client already timed out
0806 20:35:47.334654 (+ 64541us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:35:47.437731 14554 rpcz_store.cc:276] 0806 20:35:45.890718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.353549 (+1462831us) service_pool.cc:227] Handling call
0806 20:35:47.353926 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.354153 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.354182 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:47.354227 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:47.354266 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.354328 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:47.354455 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:46.325868 17015 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.280251 14475 rpcz_store.cc:276] 0806 20:35:46.203519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.278407 (+1074888us) service_pool.cc:227] Handling call
0806 20:35:47.279592 (+ 1185us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.440052 14477 rpcz_store.cc:269] 0806 20:35:45.077760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.962990 (+885230us) service_pool.cc:227] Handling call
0806 20:35:47.305394 (+1342404us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.435926 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:47.431667 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:47.454527 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1820) took 1203 ms. Trace:
I20260806 20:35:46.228353 16999 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.406493 14471 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
W20260806 20:35:47.497985 14471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1726) took 2462 ms (client timeout 1800 ms). Trace:
I20260806 20:35:47.498191 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:47.498344 14471 rpcz_store.cc:269] 0806 20:35:45.035909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.122557 (+1086648us) service_pool.cc:227] Handling call
0806 20:35:47.497904 (+1375347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.499176 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1568) took 2313 ms. Trace:
I20260806 20:35:47.499395 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:47.499728 14475 rpcz_store.cc:276] 0806 20:35:45.186163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.497842 (+2311679us) service_pool.cc:227] Handling call
0806 20:35:47.499118 (+ 1276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.500639 15178 raft_consensus.cc:2379] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:47.500962 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 354) took 1166 ms. Trace:
I20260806 20:35:47.501405 14550 rpcz_store.cc:276] 0806 20:35:46.334397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.325194 (+990797us) service_pool.cc:227] Handling call
0806 20:35:47.500860 (+175666us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.414860 16947 consensus_meta.cc:287] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.258s user 0.011s sys 0.000s
I20260806 20:35:47.282541 14549 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:47.503364 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1677) took 1317 ms. Trace:
I20260806 20:35:47.503798 14549 rpcz_store.cc:276] 0806 20:35:46.185574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.264857 (+1079283us) service_pool.cc:227] Handling call
0806 20:35:47.503310 (+238453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.352841 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:47.413164 16969 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.465724 14369 catalog_manager.cc:5495] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 3, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:47.411847 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:46.592981 16872 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.353335 16974 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.525668 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 355) took 1191 ms. Trace:
I20260806 20:35:47.526208 14555 rpcz_store.cc:276] 0806 20:35:46.334688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.306112 (+971424us) service_pool.cc:227] Handling call
0806 20:35:47.525582 (+219470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.306160 16951 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:47.460727 14632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 352) took 2386 ms (client timeout 1527 ms). Trace:
I20260806 20:35:47.419572 16935 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.353041 16855 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.335577 16942 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.471797 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:47.536145 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 357) took 1413 ms. Trace:
I20260806 20:35:47.357234 16853 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.352725 17059 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:47.458310 16908 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.590209 16908 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.488444 14631 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:47.490725 14476 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:47.432201 16962 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:46.233179 16971 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.461431 16986 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:47.497228 14633 rpcz_store.cc:276] 0806 20:35:46.228485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.413491 (+1185006us) service_pool.cc:227] Handling call
0806 20:35:47.432005 (+ 18514us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.413065 15150 connection.cc:232] RPC call timeout handler was delayed by -1.0564633189933375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17900000000000002s for a response
I20260806 20:35:47.352205 16654 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.431977 16844 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:46.611161 16848 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.516188 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:47.503229 14628 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:47.524796 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:47.525156 14627 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:47.626152 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 359) took 1301 ms. Trace:
I20260806 20:35:47.626575 14627 rpcz_store.cc:276] 0806 20:35:46.325123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.411537 (+1086414us) service_pool.cc:227] Handling call
0806 20:35:47.626074 (+214537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.488010 16946 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:47.479029 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1670) took 1581 ms. Trace:
I20260806 20:35:47.627678 14546 rpcz_store.cc:276] 0806 20:35:45.891095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.398687 (+1507592us) service_pool.cc:227] Handling call
0806 20:35:47.455423 (+ 56736us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.461079 (+ 5656us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.461110 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:47.471910 (+ 10800us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:47.471954 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.472010 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:47.472159 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.517158 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:47.628718 14469 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:47.354914 16875 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.535323 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:47.533960 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:47.632236 14554 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:47.538594 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:47.539016 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:47.539095 14626 rpcz_store.cc:276] 0806 20:35:46.117628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.325838 (+1208210us) service_pool.cc:227] Handling call
0806 20:35:47.530639 (+204801us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.546681 14369 catalog_manager.cc:5495] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 3 to 7, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131). New cstate: current_term: 7 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:47.539934 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1464) took 1215 ms. Trace:
I20260806 20:35:47.633708 14629 rpcz_store.cc:276] 0806 20:35:46.324411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.335026 (+1010615us) service_pool.cc:227] Handling call
0806 20:35:47.539854 (+204828us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.532416 14632 rpcz_store.cc:269] 0806 20:35:45.074667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.401116 (+326449us) service_pool.cc:227] Handling call
0806 20:35:47.455095 (+2053979us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.632228 14628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1815) took 1687 ms (client timeout 2086 ms). Trace:
W20260806 20:35:47.635492 14628 rpcz_store.cc:269] 0806 20:35:45.937870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.323155 (+1385285us) service_pool.cc:227] Handling call
0806 20:35:47.625214 (+302059us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.499212 16999 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.497368 16286 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:47.514060 17052 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.515275 17075 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:47.542057 16942 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.588815 15181 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 468) took 2559 ms. Trace:
I20260806 20:35:47.589123 14625 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:47.589203 14473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 324) took 1387 ms (client timeout 1790 ms). Trace:
I20260806 20:35:47.694676 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.516892 16834 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:47.587935 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:47.611544 16853 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.611954 17059 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.613493 16974 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:47.614063 16962 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.529633 16903 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:47.618692 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.544322 16935 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.491397 17002 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.625437 14470 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 8.
I20260806 20:35:47.722368 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 331) took 1238 ms. Trace:
I20260806 20:35:47.722806 14470 rpcz_store.cc:276] 0806 20:35:46.484543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.497229 (+1012686us) service_pool.cc:227] Handling call
0806 20:35:47.722307 (+225078us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.545403 17065 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:47.646919 17004 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:47.647980 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 330) took 1443 ms. Trace:
I20260806 20:35:47.732841 14477 rpcz_store.cc:276] 0806 20:35:46.204248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.496965 (+1292717us) service_pool.cc:227] Handling call
0806 20:35:47.647901 (+150936us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.609760 16947 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:47.651078 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:47.608660 16969 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:47.532869 16910 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.534662 16820 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.670730 14550 raft_consensus.cc:2396] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:47.671489 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.674154 17103 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.765956 14479 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:47.745013 15183 raft_consensus.cc:1226] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: []
W20260806 20:35:47.694600 14473 rpcz_store.cc:269] 0806 20:35:46.201850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.454557 (+1252707us) service_pool.cc:227] Handling call
0806 20:35:47.589110 (+134553us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.592127 14476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1569) took 2400 ms (client timeout 1707 ms). Trace:
W20260806 20:35:47.783296 14476 rpcz_store.cc:269] 0806 20:35:45.192521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.233595 (+1041074us) service_pool.cc:227] Handling call
0806 20:35:47.592030 (+1358435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.700215 16946 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:47.699726 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 48790.7us
I20260806 20:35:47.608654 16872 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.709452 14552 raft_consensus.cc:1226] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:47.715744 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.653887 17047 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.721856 14626 raft_consensus.cc:2396] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 20:35:47.675585 16945 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.624778 16951 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.728104 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:47.591807 17021 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:47.740917 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 561640us
I20260806 20:35:47.619977 17015 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.701717 16286 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.743774 17052 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.731251 14369 catalog_manager.cc:5495] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:47.714036 16854 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.750401 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:47.752652 16910 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:47.753162 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1805) took 3072 ms. Trace:
I20260806 20:35:47.753605 14633 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
W20260806 20:35:47.823350 14633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 356) took 1719 ms (client timeout 2282 ms). Trace:
W20260806 20:35:47.823676 14633 rpcz_store.cc:269] 0806 20:35:46.117186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.618365 (+1501179us) service_pool.cc:227] Handling call
0806 20:35:47.823294 (+204929us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.701558 15181 rpcz_store.cc:276] 0806 20:35:45.029786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.031768 (+ 1982us) service_pool.cc:227] Handling call
0806 20:35:45.076408 (+ 44640us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.334829 (+2258421us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.542421 (+207592us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b540044e154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:47.588679 (+ 46258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2014816000}
I20260806 20:35:47.686844 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:47.822535 14630 rpcz_store.cc:276] 0806 20:35:44.680987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.747636 (+ 66649us) service_pool.cc:227] Handling call
0806 20:35:44.748066 (+ 430us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.471312 (+2723246us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.735209 (+263897us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54002c2254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:47.753077 (+ 17868us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5881548544}
I20260806 20:35:47.828809 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 329) took 1625 ms. Trace:
I20260806 20:35:47.829176 14470 rpcz_store.cc:276] 0806 20:35:46.203908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.734471 (+1530563us) service_pool.cc:227] Handling call
0806 20:35:47.828748 (+ 94277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.778882 16853 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.766774 14474 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
W20260806 20:35:47.834492 14474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1570) took 2641 ms (client timeout 1560 ms). Trace:
W20260806 20:35:47.834942 14474 rpcz_store.cc:269] 0806 20:35:45.193406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.966587 (+773181us) service_pool.cc:227] Handling call
0806 20:35:47.834402 (+1867815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.734156 16974 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:47.786971 16962 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:47.787448 14479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 325) took 1579 ms (client timeout 1603 ms). Trace:
I20260806 20:35:47.797531 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 315) took 4273 ms. Trace:
I20260806 20:35:47.841359 14476 rpcz_store.cc:276] 0806 20:35:43.523704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.784021 (+4260317us) service_pool.cc:227] Handling call
0806 20:35:47.787098 (+ 3077us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.787305 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.787339 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:47.797211 (+ 9872us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:47.797272 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.797348 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:47.797469 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:47.841239 14479 rpcz_store.cc:269] 0806 20:35:46.202251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.326266 (+1124015us) service_pool.cc:227] Handling call
0806 20:35:47.781061 (+454795us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b54002319d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:47.781176 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42617600}
I20260806 20:35:47.838474 14471 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:47.697939 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:47.731974 16935 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.806097 14632 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:47.810721 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1821) took 1581 ms. Trace:
I20260806 20:35:47.846179 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1814) took 1986 ms. Trace:
I20260806 20:35:47.846134 14626 rpcz_store.cc:276] 0806 20:35:46.228907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.633503 (+1404596us) service_pool.cc:227] Handling call
0806 20:35:47.810645 (+177142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.812793 14627 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 8.
I20260806 20:35:47.847174 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 358) took 1522 ms. Trace:
I20260806 20:35:47.847453 14627 rpcz_store.cc:276] 0806 20:35:46.324760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.627383 (+1302623us) service_pool.cc:227] Handling call
0806 20:35:47.847121 (+219738us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.705929 16848 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.813683 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:47.821280 14547 raft_consensus.cc:2379] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:47.764588 17065 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.780004 16947 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.836007 14477 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:35:47.811522 17010 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:47.851863 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 309) took 4329 ms. Trace:
I20260806 20:35:47.846581 14632 rpcz_store.cc:276] 0806 20:35:45.859831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.678462 (+1818631us) service_pool.cc:227] Handling call
0806 20:35:47.846121 (+167659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.852223 14477 rpcz_store.cc:276] 0806 20:35:43.522028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.749795 (+4227767us) service_pool.cc:227] Handling call
0806 20:35:47.823440 (+ 73645us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.851624 (+ 28184us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.851812 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.845422 14629 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:47.853121 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1818) took 1737 ms. Trace:
I20260806 20:35:47.853653 14629 rpcz_store.cc:276] 0806 20:35:46.115383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.658919 (+1543536us) service_pool.cc:227] Handling call
0806 20:35:47.853070 (+194151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.773491 14554 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:47.850057 14469 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:35:47.862010 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 323) took 1665 ms (client timeout 2079 ms). Trace:
W20260806 20:35:47.862534 14469 rpcz_store.cc:269] 0806 20:35:46.196362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.493754 (+1297392us) service_pool.cc:227] Handling call
0806 20:35:47.861938 (+368184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.841219 14369 catalog_manager.cc:5495] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 4, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 4 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:47.833575 16908 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:47.883488 17030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.704s user 0.025s sys 0.050s
W20260806 20:35:47.883896 17030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.704s user 0.025s sys 0.050s
I20260806 20:35:47.882421 14473 raft_consensus.cc:1226] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 20:35:47.815946 16945 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:47.830602 15178 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 406) took 2797 ms (client timeout 1815 ms). Trace:
I20260806 20:35:47.898691 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 312) took 4375 ms. Trace:
I20260806 20:35:47.899211 14473 rpcz_store.cc:276] 0806 20:35:43.522889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.787388 (+4264499us) service_pool.cc:227] Handling call
0806 20:35:47.830311 (+ 42923us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:47.898382 (+ 68071us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.898624 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.829770 16286 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:47.783892 17103 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:47.903337 14633 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:47.911787 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 349) took 3009 ms. Trace:
I20260806 20:35:47.912171 14633 rpcz_store.cc:276] 0806 20:35:44.897058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.841694 (+2944636us) service_pool.cc:227] Handling call
0806 20:35:47.903170 (+ 61476us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.905807 (+ 2637us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.905985 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.914937 14476 raft_consensus.cc:1261] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:47.915581 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1721) took 2896 ms. Trace:
W20260806 20:35:47.915655 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 72103us
I20260806 20:35:47.915930 14476 rpcz_store.cc:276] 0806 20:35:45.018745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.868312 (+2849567us) service_pool.cc:227] Handling call
0806 20:35:47.911274 (+ 42962us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.915310 (+ 4036us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.915506 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.871141 16974 raft_consensus.cc:686] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:47.907966 14628 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:47.920902 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 466) took 2891 ms. Trace:
I20260806 20:35:47.923794 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1529) took 3826 ms. Trace:
I20260806 20:35:47.877643 16844 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:47.877434 16855 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:47.924484 14475 raft_consensus.cc:1226] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:35:48.019133 14477 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
W20260806 20:35:48.007604 14478 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.061s sys 0.110s
I20260806 20:35:48.009853 14479 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260806 20:35:48.010447 14624 consensus_meta.cc:287] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.595s user 0.005s sys 0.006s
W20260806 20:35:48.021682 14478 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.062s sys 0.110s
I20260806 20:35:47.970263 14471 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:48.013713 14547 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:48.021993 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 322) took 3983 ms. Trace:
I20260806 20:35:48.014784 14469 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:47.876355 16962 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:48.020584 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 318) took 4221 ms. Trace:
I20260806 20:35:48.021170 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1722) took 3002 ms. Trace:
I20260806 20:35:48.014046 14549 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:48.031301 14477 rpcz_store.cc:276] 0806 20:35:45.019194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.882987 (+2863793us) service_pool.cc:227] Handling call
0806 20:35:48.013496 (+130509us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.021025 (+ 7529us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.021131 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.031653 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 350) took 3123 ms. Trace:
I20260806 20:35:48.022593 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 348) took 3137 ms. Trace:
I20260806 20:35:48.022498 14479 rpcz_store.cc:276] 0806 20:35:44.038596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.876575 (+3837979us) service_pool.cc:227] Handling call
0806 20:35:47.915222 (+ 38647us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.021761 (+106539us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.021940 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.028400 14473 raft_consensus.cc:1261] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:35:48.016242 14552 rpcz_store.cc:276] 0806 20:35:44.096924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.097281 (+ 357us) service_pool.cc:227] Handling call
0806 20:35:45.307548 (+1210267us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:35:45.307812 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.812525 (+2504713us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.910746 (+ 98221us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540018e2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:47.923716 (+ 12970us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5347606784}
I20260806 20:35:48.032202 14630 rpcz_store.cc:276] 0806 20:35:44.897367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.842070 (+2944703us) service_pool.cc:227] Handling call
0806 20:35:47.903219 (+ 61149us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.020786 (+117567us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:48.020806 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:48.020831 (+ 25us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:48.020851 (+ 20us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.020888 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.020959 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.032910 14628 rpcz_store.cc:276] 0806 20:35:44.876007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.709190 (+2833183us) service_pool.cc:227] Handling call
0806 20:35:47.873206 (+164016us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.012979 (+139773us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.013192 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.021903 14624 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:48.022585 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 326) took 1820 ms. Trace:
I20260806 20:35:48.034144 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 327) took 1831 ms. Trace:
I20260806 20:35:48.035887 14473 rpcz_store.cc:276] 0806 20:35:46.203116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.900101 (+1696985us) service_pool.cc:227] Handling call
0806 20:35:48.011488 (+111387us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.033895 (+ 22407us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.034093 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.030402 14475 rpcz_store.cc:276] 0806 20:35:43.799657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.827882 (+4028225us) service_pool.cc:227] Handling call
0806 20:35:47.881824 (+ 53942us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:48.020270 (+138446us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.020510 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.032639 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1724) took 3008 ms. Trace:
I20260806 20:35:48.037678 14469 rpcz_store.cc:276] 0806 20:35:45.024809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.890126 (+2865317us) service_pool.cc:227] Handling call
0806 20:35:48.011992 (+121866us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.032367 (+ 20375us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.032582 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.029057 14625 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:48.039960 14471 rpcz_store.cc:276] 0806 20:35:46.202641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.888254 (+1685613us) service_pool.cc:227] Handling call
0806 20:35:47.917252 (+ 28998us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.022338 (+105086us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.022514 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:47.926187 16971 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.022105 14369 catalog_manager.cc:5495] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 3, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:48.034081 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1813) took 2171 ms. Trace:
I20260806 20:35:48.042743 14629 rpcz_store.cc:276] 0806 20:35:45.859017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.874326 (+2015309us) service_pool.cc:227] Handling call
0806 20:35:47.908293 (+ 33967us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.973094 (+ 64801us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:48.004893 (+ 31799us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:48.014649 (+ 9756us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:48.029902 (+ 15253us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.029976 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.030093 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:48.044917 14624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1463) took 1711 ms (client timeout 2038 ms). Trace:
W20260806 20:35:48.044858 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1817) took 2022 ms (client timeout 1571 ms). Trace:
W20260806 20:35:47.898262 15178 rpcz_store.cc:269] 0806 20:35:45.028648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.032827 (+ 4179us) service_pool.cc:227] Handling call
0806 20:35:47.787708 (+2754881us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54006504d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:47.822009 (+ 34301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3013911936}
W20260806 20:35:48.045751 14625 rpcz_store.cc:269] 0806 20:35:46.018055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.322846 (+1304791us) service_pool.cc:227] Handling call
0806 20:35:48.039870 (+717024us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:48.045585 14624 rpcz_store.cc:269] 0806 20:35:46.324061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.413660 (+1089599us) service_pool.cc:227] Handling call
0806 20:35:48.035487 (+621827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.049288 14624 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 20:35:47.994503 17059 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.040503 16654 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.055013 14552 raft_consensus.cc:1226] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:48.056479 16951 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:48.057157 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.011s sys 0.032s
W20260806 20:35:48.057672 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.944s user 0.011s sys 0.032s
I20260806 20:35:48.060871 14627 raft_consensus.cc:1261] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:48.061564 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1809) took 2855 ms. Trace:
I20260806 20:35:48.061916 14627 rpcz_store.cc:276] 0806 20:35:45.205821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.876682 (+2670861us) service_pool.cc:227] Handling call
0806 20:35:48.013598 (+136916us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.061303 (+ 47705us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.061515 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.032575 17002 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:48.063985 16286 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.064152 16820 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.069094 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1717) took 4744 ms. Trace:
I20260806 20:35:48.069666 14478 rpcz_store.cc:276] 0806 20:35:43.324849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:44.109052 (+784203us) service_pool.cc:227] Handling call
0806 20:35:44.135698 (+ 26646us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:44.135901 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:44.904895 (+768994us) log.cc:842] Serialized 14 byte log entry
0806 20:35:48.047329 (+3142434us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:48.068583 (+ 21254us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:48.068674 (+ 91us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.068784 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.068992 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":38735,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":92770,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":20268,"thread_start_us":3784256,"threads_started":3,"wal-append.queue_time_us":949086,"wal-append.run_cpu_time_us":2664,"wal-append.run_wall_time_us":1146504}
I20260806 20:35:48.076362 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1462) took 1726 ms. Trace:
I20260806 20:35:48.048797 16875 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:48.016125 15183 rpcz_store.cc:276] 0806 20:35:45.029214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:45.031739 (+ 2525us) service_pool.cc:227] Handling call
0806 20:35:45.050936 (+ 19197us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.809844 (+2758908us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.920620 (+110776us) spinlock_profiling.cc:243] Waited 2.66 s on lock 0x7b5400651654. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:47.920839 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6123905920}
I20260806 20:35:48.078595 14632 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:48.078500 14624 rpcz_store.cc:276] 0806 20:35:46.323699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:48.048591 (+1724892us) service_pool.cc:227] Handling call
0806 20:35:48.049034 (+ 443us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:48.050011 (+ 977us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.050295 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.028496 17046 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.053402 14369 catalog_manager.cc:5495] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:47.900931 17010 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:48.081840 14520 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.008s sys 0.033s
W20260806 20:35:48.096742 14520 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.958s user 0.009s sys 0.034s
I20260806 20:35:48.092566 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1810) took 2810 ms. Trace:
I20260806 20:35:48.097766 14632 rpcz_store.cc:276] 0806 20:35:45.282258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.883014 (+2600756us) service_pool.cc:227] Handling call
0806 20:35:48.011405 (+128391us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.092202 (+ 80797us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.092492 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.046752 16942 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:48.095371 14443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.001s sys 0.041s
I20260806 20:35:48.101073 14369 catalog_manager.cc:5495] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
W20260806 20:35:48.096560 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.043s sys 0.060s
W20260806 20:35:48.120949 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.930s user 0.044s sys 0.060s
W20260806 20:35:48.111817 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 70760.7us
W20260806 20:35:48.120539 14443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s user 0.001s sys 0.043s
I20260806 20:35:48.102447 16946 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.103024 16872 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.102373 17002 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:48.154461 17104 log.cc:924] Time spent T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.425s user 0.001s sys 0.001s
I20260806 20:35:48.156111 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1533) took 3181 ms. Trace:
I20260806 20:35:48.156487 14551 rpcz_store.cc:276] 0806 20:35:44.974587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:46.091603 (+1117016us) service_pool.cc:227] Handling call
0806 20:35:46.092106 (+ 503us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:46.092752 (+ 646us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:46.092793 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:46.093333 (+ 540us) log.cc:842] Serialized 26 byte log entry
0806 20:35:47.280981 (+1187648us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:47.281036 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:47.281153 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:48.153704 (+872551us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:48.155812 (+ 2108us) raft_consensus.cc:1671] finished
0806 20:35:48.155832 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.156050 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10017,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":28478,"spinlock_wait_cycles":152925440,"thread_start_us":94704,"threads_started":1,"wal-append.queue_time_us":1532088}
I20260806 20:35:48.157835 15153 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:48.158619 17059 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:48.159229 17059 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.160338 16286 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:48.160866 16286 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:48.161304 16286 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:48.161862 17059 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.162592 17059 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:48.163185 17059 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.166765 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1675) took 2209 ms. Trace:
I20260806 20:35:48.167253 14555 rpcz_store.cc:276] 0806 20:35:45.957922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.527051 (+1569129us) service_pool.cc:227] Handling call
0806 20:35:47.594479 (+ 67428us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.912366 (+317887us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.912396 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:47.912434 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:47.926530 (+ 14096us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.166476 (+239946us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54000a8a54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:48.166536 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.166694 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679584640}
I20260806 20:35:48.149952 17065 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.174019 17015 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.126205 16848 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:48.130204 14369 catalog_manager.cc:5495] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: HEALTHY } } }
I20260806 20:35:48.132035 17103 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.134394 17021 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.134248 16853 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.135895 17010 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.209502 14551 raft_consensus.cc:1261] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:48.210990 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:48.181834 17046 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.212700 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:48.212441 17046 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:48.213812 14552 raft_consensus.cc:2396] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:48.136749 16945 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:48.144076 14631 consensus_meta.cc:287] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.528s user 0.008s sys 0.000s
I20260806 20:35:48.214948 14631 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:48.215245 17046 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:48.215579 14631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 353) took 2356 ms (client timeout 1603 ms). Trace:
W20260806 20:35:48.216045 14631 rpcz_store.cc:269] 0806 20:35:45.859018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.274661 (+1415643us) service_pool.cc:227] Handling call
0806 20:35:48.215498 (+940837us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.217165 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.217991 14631 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:48.147879 16947 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.148721 17052 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:48.180267 16935 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.188275 16946 consensus_queue.cc:1046] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
I20260806 20:35:48.175001 17002 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.221450 16955 raft_consensus.cc:3032] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:48.221808 16955 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:48.222657 16955 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:48.223223 17002 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.223965 16962 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:48.192724 16910 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:48.225246 16910 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.226410 16946 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:35:48.227754 16910 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.228778 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:48.187711 14441 connection.cc:232] RPC call timeout handler was delayed by -1.045514561003074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
I20260806 20:35:48.196769 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:48.201709 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:48.202306 17107 log.cc:924] Time spent T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.552s user 0.000s sys 0.005s
I20260806 20:35:48.205822 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:48.242486 14550 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:48.224045 16955 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:48.182860 16942 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.233553 14624 raft_consensus.cc:2362] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6. Re-sending same reply.
I20260806 20:35:48.127338 14629 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 20:35:48.244623 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 44723.8us
I20260806 20:35:48.244705 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1461) took 1921 ms. Trace:
I20260806 20:35:48.245766 14629 rpcz_store.cc:276] 0806 20:35:46.323292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:48.043895 (+1720603us) service_pool.cc:227] Handling call
0806 20:35:48.083851 (+ 39956us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.244325 (+160474us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.244645 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:48.246570 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:48.247545 14632 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:48.256685 14549 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:48.258519 14369 catalog_manager.cc:5495] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:48.270445 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:48.257586 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.905s user 0.019s sys 0.027s
W20260806 20:35:48.273470 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.921s user 0.020s sys 0.027s
I20260806 20:35:48.275426 14596 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:48.277359 14369 catalog_manager.cc:5495] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 4, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 4 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
W20260806 20:35:48.297910 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 2.19599e+06us
I20260806 20:35:48.233711 17015 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:48.198971 14555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:48.249172 17010 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
W20260806 20:35:48.300984 14444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.007s sys 0.040s
I20260806 20:35:48.304400 14369 catalog_manager.cc:5495] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
W20260806 20:35:48.309243 14442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.025s sys 0.071s
I20260806 20:35:48.171523 16974 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:48.325826 14444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.903s user 0.007s sys 0.040s
I20260806 20:35:48.301545 14555 raft_consensus.cc:595] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:48.329500 14442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.855s user 0.025s sys 0.072s
W20260806 20:35:48.324677 14548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.014s sys 0.037s
W20260806 20:35:48.332600 15151 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.023s sys 0.054s
W20260806 20:35:48.333004 15151 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.849s user 0.023s sys 0.054s
W20260806 20:35:48.332147 14548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.909s user 0.015s sys 0.037s
I20260806 20:35:48.340294 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:48.334070 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 58757.7us
I20260806 20:35:48.259722 16942 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.345852 14442 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:48.338922 14369 catalog_manager.cc:5495] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d reported cstate change: term changed from 1 to 4, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 4 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:48.372095 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.390357 14595 leader_election.cc:398] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:48.391010 14595 leader_election.cc:401] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 20:35:48.419272 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.420576 14631 raft_consensus.cc:2396] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:48.429339 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:48.429998 15183 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:48.433823 15181 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:35:48.434654 15181 raft_consensus.cc:595] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:48.435112 15181 raft_consensus.cc:603] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:35:48.434023 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 88264.6us
W20260806 20:35:48.436609 14521 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.011s sys 0.007s
I20260806 20:35:48.436697 16935 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:48.437129 14521 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.673s user 0.011s sys 0.007s
I20260806 20:35:48.437209 16935 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:48.437471 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 714324us
I20260806 20:35:48.437624 16935 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:48.438205 14442 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:48.439116 17052 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.439800 17111 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:48.439673 17052 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.442211 17052 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.443754 14442 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:48.443846 16286 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:48.444527 14598 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.992s user 0.027s sys 0.022s
W20260806 20:35:48.444952 14598 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.171s user 0.027s sys 0.022s
I20260806 20:35:48.370292 16854 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.445529 16854 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.447708 16854 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.448222 16854 raft_consensus.cc:2741] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:35:48.450758 16947 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:48.452026 14442 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:48.454633 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1674) took 2562 ms. Trace:
I20260806 20:35:48.455505 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:48.459589 14625 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:48.456184 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.373975 17059 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.335083 17010 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.469099 14625 raft_consensus.cc:2396] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:35:48.465188 14548 rpcz_store.cc:276] 0806 20:35:45.893783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.420997 (+1527214us) service_pool.cc:227] Handling call
0806 20:35:47.421492 (+ 495us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:47.422090 (+ 598us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.422123 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:47.422846 (+ 723us) log.cc:842] Serialized 52 byte log entry
0806 20:35:48.336479 (+913633us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:48.336517 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.336582 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:48.454428 (+117846us) raft_consensus.cc:1671] finished
0806 20:35:48.454447 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.454577 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14232,"raft.queue_time_us":9629,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":39390,"thread_start_us":913121,"threads_started":1,"wal-append.queue_time_us":1029558}
I20260806 20:35:48.479385 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.469807 17110 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:48.481700 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.482632 14627 raft_consensus.cc:2379] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:48.422017 17114 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:48.342240 17015 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:48.486505 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:48.491253 14553 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.020s sys 0.020s
W20260806 20:35:48.491604 14553 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.609s user 0.020s sys 0.020s
I20260806 20:35:48.496608 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.497752 14627 raft_consensus.cc:2396] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:35:48.500097 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.503862 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:48.505882 14518 connection.cc:232] RPC call timeout handler was delayed by -1.4918299840064719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
W20260806 20:35:48.506371 14518 connection.cc:232] RPC call timeout handler was delayed by -1.1246299839986023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
I20260806 20:35:48.509006 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.509829 14471 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:48.519788 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.523074 15181 raft_consensus.cc:1261] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260806 20:35:48.491472 16854 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:48.523383 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:48.491007 16910 consensus_queue.cc:1035] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 20:35:48.524789 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.523447 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:48.488049 17110 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:48.525806 14469 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:48.526217 14518 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:48.520934 14631 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:48.495107 17052 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.532615 14475 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:48.528400 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:48.529696 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 139561us
I20260806 20:35:48.528900 14444 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:48.503391 17046 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.537204 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.545986 14473 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:35:48.546187 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:48.546829 14548 raft_consensus.cc:595] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:48.536598 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.547262 14548 raft_consensus.cc:603] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:48.523505 15181 pending_rounds.cc:76] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Aborting all ops after (but not including) 4
I20260806 20:35:48.510713 17002 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.514576 17103 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.610466 17103 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:48.535507 16969 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.034s sys 0.028s
I20260806 20:35:48.555606 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.607527 14442 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:48.557693 17109 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:48.609807 15182 raft_consensus.cc:1226] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
W20260806 20:35:48.613472 14554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.020s sys 0.043s
I20260806 20:35:48.613867 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.628790 14479 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
W20260806 20:35:48.618141 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 92003.2us
I20260806 20:35:48.620189 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:48.629937 14554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s user 0.020s sys 0.043s
I20260806 20:35:48.535727 16286 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.530391 17110 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:48.631935 14630 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.505s user 0.020s sys 0.011s
I20260806 20:35:48.630102 14477 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:48.617748 14475 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:48.533819 16947 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.534134 17059 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:48.627401 14469 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:48.628672 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.590286 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.548223 16854 raft_consensus.cc:686] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:48.621270 14546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.014s sys 0.027s
I20260806 20:35:48.551712 15178 raft_consensus.cc:1226] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:48.635066 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:48.617332 16969 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.693s user 0.034s sys 0.028s
I20260806 20:35:48.631589 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:48.553417 15181 pending_rounds.cc:97] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260806 20:35:48.639853 14595 leader_election.cc:398] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:48.651620 14595 leader_election.cc:401] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260806 20:35:48.642913 14546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.014s sys 0.027s
I20260806 20:35:48.641884 14631 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:48.645298 14476 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
W20260806 20:35:48.649371 14626 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.002s sys 0.028s
W20260806 20:35:48.644376 14474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.010s sys 0.037s
W20260806 20:35:48.654076 14626 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.540s user 0.002s sys 0.028s
I20260806 20:35:48.654304 14518 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:48.654363 14474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s user 0.010s sys 0.037s
I20260806 20:35:48.654107 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:48.657518 14475 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:48.646922 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:48.616882 16908 consensus_queue.cc:1046] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260806 20:35:48.553222 17046 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.524710 17114 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.554570 17052 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:48.655431 16654 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.019s sys 0.032s
W20260806 20:35:48.690308 16654 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.543s user 0.020s sys 0.032s
I20260806 20:35:48.690398 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.670169 17108 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:48.691131 14476 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:48.677472 16854 consensus_queue.cc:237] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.694999 17065 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:48.697741 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:48.698805 15179 raft_consensus.cc:2396] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260806 20:35:48.699800 15187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:35:48.700480 15187 raft_consensus.cc:595] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:48.700827 15187 raft_consensus.cc:603] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:48.701632 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.612576 17116 consensus_queue.cc:1046] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260806 20:35:48.660269 15181 raft_consensus.cc:2866] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:48.654932 16942 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.739730 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.740659 14625 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:48.648262 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:48.653194 17002 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.651873 17059 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.757447 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.652242 16935 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:48.757237 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 382912us
I20260806 20:35:48.658915 16947 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:48.760257 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:48.721206 16853 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:48.765836 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 150093us
I20260806 20:35:48.750214 17103 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:49.130151 15183 consensus_meta.cc:287] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.700s user 0.013s sys 0.000s
I20260806 20:35:48.817787 14625 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:49.134354 15183 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:48.830189 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:48.989519 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 351) took 3931 ms. Trace:
I20260806 20:35:48.777664 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.843225 14478 raft_consensus.cc:2379] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:48.859035 14548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:49.091710 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:48.887853 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:49.136818 14548 raft_consensus.cc:595] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:48.757913 16942 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:48.933946 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:48.934850 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.143463 14473 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
W20260806 20:35:48.868485 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 261024us
I20260806 20:35:48.937153 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:48.844743 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 320173us
I20260806 20:35:49.136472 14626 rpcz_store.cc:276] 0806 20:35:45.058538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.859173 (+2800635us) service_pool.cc:227] Handling call
0806 20:35:48.010800 (+151627us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.033914 (+ 23114us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:48.660187 (+626273us) log.cc:842] Serialized 14 byte log entry
0806 20:35:48.969130 (+308943us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:48.969183 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:48.989165 (+ 19982us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.989267 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.989438 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137112,"prepare.queue_time_us":694429,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":126608,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":24161,"thread_start_us":709948,"threads_started":2}
I20260806 20:35:49.167322 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:49.000222 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1816) took 2983 ms. Trace:
I20260806 20:35:49.137478 14548 raft_consensus.cc:603] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:48.888549 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 352) took 2554 ms. Trace:
W20260806 20:35:48.777743 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 134793us
I20260806 20:35:49.139698 16286 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
W20260806 20:35:49.136965 15153 leader_election.cc:334] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209): Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.527s (SENT)
I20260806 20:35:49.177795 15153 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.147218 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.148811 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:49.149187 17143 raft_consensus.cc:3032] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:48.913545 17015 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:49.126103 17052 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:49.136986 15186 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:49.151489 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.186556 14555 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:49.152537 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.152779 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.187116 14595 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:49.153548 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.156728 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:49.157325 17138 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
I20260806 20:35:49.158303 16855 raft_consensus.cc:3032] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:49.008652 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 355) took 3005 ms. Trace:
I20260806 20:35:49.083772 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:49.048022 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.137270 14479 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:49.143819 14471 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 2.
W20260806 20:35:48.920315 17128 log.cc:924] Time spent T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.242s user 0.000s sys 0.002s
I20260806 20:35:49.168486 14628 rpcz_store.cc:276] 0806 20:35:46.017251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:48.034058 (+2016807us) service_pool.cc:227] Handling call
0806 20:35:48.081016 (+ 46958us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.125005 (+ 43989us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:35:48.626835 (+501830us) log.cc:842] Serialized 14 byte log entry
0806 20:35:48.999595 (+372760us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:48.999667 (+ 72us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:48.999737 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.999950 (+ 213us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:49.000090 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1628,"prepare.queue_time_us":757375,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":20130,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":18895,"thread_start_us":831018,"threads_started":2}
I20260806 20:35:49.174952 14553 rpcz_store.cc:276] 0806 20:35:46.333707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.790125 (+1456418us) service_pool.cc:227] Handling call
0806 20:35:47.807240 (+ 17115us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:47.829569 (+ 22329us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:48.498646 (+669077us) log.cc:842] Serialized 14 byte log entry
0806 20:35:48.888105 (+389459us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:48.888181 (+ 76us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:48.888251 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.888330 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:48.888479 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":844754,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":942,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":6224,"thread_start_us":871455,"threads_started":2,"wal-append.queue_time_us":559954}
I20260806 20:35:49.178769 15185 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:49.178864 14473 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:48.933265 16947 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.929651 17059 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.815163 17010 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:48.763705 17112 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:49.237054 14444 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:49.237805 16955 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.988s user 0.012s sys 0.000s
W20260806 20:35:49.238108 14475 consensus_meta.cc:287] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.552s user 0.010s sys 0.000s
I20260806 20:35:49.238356 14444 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:49.187630 14549 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:35:48.934401 17114 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:49.084317 16935 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:49.063012 17046 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:48.914261 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:48.844877 14596 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:48.933086 17002 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:49.192600 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.251754 14626 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:49.196707 15187 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:49.192818 14630 rpcz_store.cc:276] 0806 20:35:46.031834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:48.033547 (+2001713us) service_pool.cc:227] Handling call
0806 20:35:48.082081 (+ 48534us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:48.095383 (+ 13302us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:48.634739 (+539356us) log.cc:842] Serialized 14 byte log entry
0806 20:35:49.007835 (+373096us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:49.007968 (+ 133us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:49.008117 (+ 149us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:49.008261 (+ 144us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:49.008551 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16016,"prepare.queue_time_us":734465,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":19460,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":35795,"thread_start_us":850379,"threads_started":2,"wal-append.queue_time_us":286787}
I20260806 20:35:49.200428 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 353) took 2866 ms. Trace:
I20260806 20:35:49.253083 14546 rpcz_store.cc:276] 0806 20:35:46.334064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.839952 (+1505888us) service_pool.cc:227] Handling call
0806 20:35:47.872162 (+ 32210us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:47.911587 (+ 39425us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.911617 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:47.995716 (+ 84099us) log.cc:842] Serialized 26 byte log entry
0806 20:35:48.652320 (+656604us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:48.652359 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.652425 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:49.200195 (+547770us) raft_consensus.cc:1671] finished
0806 20:35:49.200214 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:49.200369 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22672,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":51093,"thread_start_us":566772,"threads_started":1,"wal-append.queue_time_us":1167003}
I20260806 20:35:49.200956 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.201246 14552 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:49.203178 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.207718 17015 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:49.182076 17143 raft_consensus.cc:729] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:49.212987 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.213748 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.214303 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260806 20:35:49.287442 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 470621us
I20260806 20:35:49.192607 16855 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:49.218339 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:49.219117 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.219637 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.220489 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:49.221240 17129 log.cc:924] Time spent T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260806 20:35:49.224141 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.233307 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.238478 14475 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:35:49.291255 14478 raft_consensus.cc:2362] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5. Re-sending same reply.
I20260806 20:35:49.239657 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.153273 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.244001 14479 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:49.242300 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:49.249161 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.235168 15178 raft_consensus.cc:2379] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:49.296036 14548 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:49.194079 16942 raft_consensus.cc:686] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:49.300050 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 122338us
I20260806 20:35:49.280175 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.286242 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.287560 14471 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:35:49.205717 17103 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:49.285602 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 131240us
I20260806 20:35:49.303417 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.291853 14550 raft_consensus.cc:2396] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260806 20:35:49.292688 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 120851us
I20260806 20:35:49.290526 14629 raft_consensus.cc:2362] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5. Re-sending same reply.
I20260806 20:35:49.295034 14551 raft_consensus.cc:2396] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:35:49.249907 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.305614 14628 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:35:49.299213 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:49.299438 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 57245.9us
I20260806 20:35:49.256902 14477 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:49.301689 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.307710 14626 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:49.300743 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:49.302415 14624 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:49.304129 14546 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:49.250897 17052 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:49.308108 14444 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:49.310539 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.311374 14552 raft_consensus.cc:2396] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:35:49.311560 14444 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.290558 17142 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:35:49.317951 14625 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:35:49.320511 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:49.298370 17113 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.009s sys 0.005s
I20260806 20:35:49.310263 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:35:49.323149 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.326066 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.327503 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.327917 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.328567 16947 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:49.328826 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.329473 17158 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:49.329927 17158 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:49.330309 17158 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:49.330503 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:49.329011 16947 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:49.329641 14555 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:49.333050 14546 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:49.295024 17002 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:49.301302 16854 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.000s sys 0.012s
I20260806 20:35:49.361725 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.362519 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:49.310575 17144 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260806 20:35:49.363617 14624 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:49.321300 15182 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:49.366506 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.367206 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:49.376914 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:49.380699 14476 raft_consensus.cc:2362] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6. Re-sending same reply.
I20260806 20:35:49.385298 14475 raft_consensus.cc:2362] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5. Re-sending same reply.
I20260806 20:35:49.367761 14444 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.368690 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.368783 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.361855 16947 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:49.369560 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.362263 17002 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:49.372861 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.365896 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.290628 16962 consensus_queue.cc:1046] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260806 20:35:49.389186 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.390081 14550 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:49.390699 15153 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:49.394017 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 254310us
I20260806 20:35:49.394270 14632 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:49.387596 14630 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:49.413751 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:49.337416 17113 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.640s user 0.009s sys 0.006s
I20260806 20:35:49.296422 16969 consensus_queue.cc:1035] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
W20260806 20:35:49.426258 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 63247.7us
W20260806 20:35:49.425179 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 121302us
I20260806 20:35:49.427906 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:49.290346 16908 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260806 20:35:49.434778 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:49.434778 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:49.436058 14549 raft_consensus.cc:2396] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:35:49.429446 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 320) took 5592 ms. Trace:
W20260806 20:35:49.429487 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 61853.1us
I20260806 20:35:49.436939 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.439318 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:49.319621 16935 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:49.318625 17015 raft_consensus.cc:686] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:35:49.381330 16854 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.000s sys 0.024s
I20260806 20:35:49.432170 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:49.420758 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 83958.1us
I20260806 20:35:49.434916 14469 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:49.448017 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 93760.6us
I20260806 20:35:49.435997 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:49.435308 14596 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.437789 14478 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:49.388968 17002 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:49.446578 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.436774 14474 rpcz_store.cc:276] 0806 20:35:43.837037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.835680 (+3998643us) service_pool.cc:227] Handling call
0806 20:35:47.881733 (+ 46053us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:47.921310 (+ 39577us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:48.013390 (+ 92080us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:48.039527 (+ 26137us) log.cc:842] Serialized 26 byte log entry
0806 20:35:48.655299 (+615772us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:48.655372 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.655516 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:49.420675 (+765159us) raft_consensus.cc:1671] finished
0806 20:35:49.420705 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:49.429317 (+ 8612us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7882,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":5486,"thread_start_us":557907,"threads_started":1,"wal-append.queue_time_us":629684}
I20260806 20:35:49.451290 14632 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:49.451588 14476 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 8.
I20260806 20:35:49.452291 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.453162 14476 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:49.458621 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1543) took 1979 ms. Trace:
I20260806 20:35:49.459200 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.461494 15151 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.464152 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.464512 14444 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.467691 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.324100 17103 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:49.473711 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.475441 14629 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:49.473050 14630 raft_consensus.cc:2396] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260806 20:35:49.473146 14471 pending_rounds.cc:76] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 3
I20260806 20:35:49.476082 14471 pending_rounds.cc:97] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:35:49.436982 17142 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:49.477095 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.477983 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.483721 14469 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:49.476401 14471 raft_consensus.cc:2866] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:35:49.480288 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 45701.6us
I20260806 20:35:49.459070 14547 rpcz_store.cc:276] 0806 20:35:47.479198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:48.072541 (+593343us) service_pool.cc:227] Handling call
0806 20:35:48.072906 (+ 365us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:48.073389 (+ 483us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:48.073419 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:48.156415 (+ 82996us) spinlock_profiling.cc:243] Waited 4.7 ms on lock 0x7b2c00000238. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:48.156895 (+ 480us) log.cc:842] Serialized 26 byte log entry
0806 20:35:48.533532 (+376637us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:48.533571 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.533644 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:49.450569 (+916925us) raft_consensus.cc:1671] finished
0806 20:35:49.450586 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:49.458534 (+ 7948us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2512,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":155,"spinlock_wait_cycles":10815360,"thread_start_us":376250,"threads_started":1,"wal-append.queue_time_us":851954}
I20260806 20:35:49.486609 14549 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:49.487619 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1476) took 1030 ms. Trace:
I20260806 20:35:49.490717 15151 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:49.493800 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 119195us
I20260806 20:35:49.487990 14630 rpcz_store.cc:276] 0806 20:35:48.457299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.435728 (+978429us) service_pool.cc:227] Handling call
0806 20:35:49.487558 (+ 51830us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:49.458271 17112 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:49.499732 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.509889 14631 raft_consensus.cc:2396] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 20:35:49.542507 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 57639.8us
I20260806 20:35:49.546727 14518 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:49.489955 17142 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:49.548542 14596 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.552806 15151 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:49.559690 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.555302 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 319) took 5752 ms. Trace:
I20260806 20:35:49.562294 14470 rpcz_store.cc:276] 0806 20:35:43.803949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.837274 (+4033325us) service_pool.cc:227] Handling call
0806 20:35:47.909910 (+ 72636us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:48.015687 (+105777us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:48.524427 (+508740us) log.cc:842] Serialized 14 byte log entry
0806 20:35:49.129902 (+605475us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:49.168879 (+ 38977us) log.cc:842] Serialized 26 byte log entry
0806 20:35:49.169027 (+ 148us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:49.169058 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:49.169116 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:49.555041 (+385925us) raft_consensus.cc:1671] finished
0806 20:35:49.555060 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:49.555212 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106646,"prepare.queue_time_us":807454,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":396623,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":24456,"thread_start_us":611614,"threads_started":2,"wal-append.queue_time_us":512558}
I20260806 20:35:49.499114 16854 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:49.576543 14630 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
I20260806 20:35:49.499809 17112 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:49.582085 14444 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.582388 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.553761 17160 raft_consensus.cc:2726] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:49.554277 17119 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:49.549897 17162 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:49.610001 14631 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
W20260806 20:35:49.609730 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 231722us
I20260806 20:35:49.610523 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.569691 17142 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:49.612726 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1478) took 1154 ms. Trace:
I20260806 20:35:49.616529 14631 rpcz_store.cc:276] 0806 20:35:48.457898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.552584 (+1094686us) service_pool.cc:227] Handling call
0806 20:35:49.612646 (+ 60062us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:49.583753 16854 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:49.582619 16910 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:49.594201 17140 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:49.596983 17112 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:49.489074 16855 consensus_queue.cc:260] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:49.610118 17133 log.cc:924] Time spent T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.327s user 0.000s sys 0.001s
I20260806 20:35:49.627784 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.633383 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.633496 14630 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:49.634183 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1484) took 1018 ms. Trace:
I20260806 20:35:49.634297 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1481) took 1161 ms. Trace:
I20260806 20:35:49.634505 14630 rpcz_store.cc:276] 0806 20:35:48.618603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.627417 (+1008814us) service_pool.cc:227] Handling call
0806 20:35:49.634133 (+ 6716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:49.634654 14631 rpcz_store.cc:276] 0806 20:35:48.473197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.617368 (+1144171us) service_pool.cc:227] Handling call
0806 20:35:49.634239 (+ 16871us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:49.635628 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.636204 14444 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:49.553138 17166 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:35:49.633294 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 86647us
I20260806 20:35:49.666275 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:49.669479 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 113613us
I20260806 20:35:49.621912 16854 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:49.671083 14518 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:49.619422 17165 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:35:49.545135 17137 consensus_queue.cc:1035] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.127s
I20260806 20:35:49.484414 16844 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260806 20:35:49.426220 17065 consensus_queue.cc:1046] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260806 20:35:49.620949 16910 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:49.685415 14630 raft_consensus.cc:2379] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:49.547991 16942 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:49.627676 17140 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:49.690367 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.664151 16935 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:49.706595 14595 connection.cc:232] RPC call timeout handler was delayed by -4.3927105779876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19890000000000002s for a response
W20260806 20:35:49.681030 17157 log.cc:924] Time spent T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
W20260806 20:35:49.675299 17148 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
W20260806 20:35:49.674923 17110 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.022s user 0.013s sys 0.000s
I20260806 20:35:49.723814 14518 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:49.713344 14624 raft_consensus.cc:2454] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:49.721382 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.718035 14631 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:49.671964 17170 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:49.628297 17119 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:49.617110 17015 consensus_queue.cc:237] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:49.713339 16910 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:49.741796 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.734612 17170 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:49.722474 17140 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:49.725625 16935 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:49.752056 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1824) took 1128 ms. Trace:
I20260806 20:35:49.764825 14625 rpcz_store.cc:276] 0806 20:35:48.624159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:48.748880 (+124721us) service_pool.cc:227] Handling call
0806 20:35:49.751857 (+1002977us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b540011fdd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:49.751987 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51254016}
I20260806 20:35:49.746588 14474 raft_consensus.cc:1261] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:49.751631 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:49.766426 14469 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:49.776484 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.776603 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.777890 14479 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260806 20:35:49.777916 14625 raft_consensus.cc:2396] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:35:49.779175 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 354) took 3834 ms. Trace:
W20260806 20:35:49.779613 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 105414us
I20260806 20:35:49.779752 14633 rpcz_store.cc:276] 0806 20:35:45.945402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.913021 (+1967619us) service_pool.cc:227] Handling call
0806 20:35:47.924967 (+ 11946us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:48.052764 (+127797us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:48.052809 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:48.121001 (+ 68192us) log.cc:842] Serialized 26 byte log entry
0806 20:35:48.642247 (+521246us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:48.642301 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.642389 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:49.778931 (+1136542us) raft_consensus.cc:1671] finished
0806 20:35:49.778957 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:49.779108 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26771,"raft.queue_time_us":20748,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":28285,"thread_start_us":459578,"threads_started":1,"wal-append.queue_time_us":1009237}
I20260806 20:35:49.781083 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.736856 17175 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:49.778329 14479 pending_rounds.cc:76] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 1
I20260806 20:35:49.782697 14479 pending_rounds.cc:97] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260806 20:35:49.783039 14479 raft_consensus.cc:2866] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:49.784981 14624 raft_consensus.cc:2396] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 20:35:49.785801 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.786836 14476 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:49.778306 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 134704us
I20260806 20:35:49.796274 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.714224 17165 consensus_queue.cc:1035] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
W20260806 20:35:49.797824 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 215526us
W20260806 20:35:49.799129 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 89505.9us
I20260806 20:35:49.763464 17170 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:49.767908 16935 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:49.801287 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.737149 17110 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:49.798200 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.803434 14476 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:49.805562 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.808193 14624 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:49.811053 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.812561 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.787904 17103 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:49.790311 17175 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:49.852068 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1479) took 1361 ms. Trace:
I20260806 20:35:49.844966 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:49.945268 14625 rpcz_store.cc:276] 0806 20:35:48.458173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.812255 (+1354082us) service_pool.cc:227] Handling call
0806 20:35:49.819895 (+ 7640us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:49.796015 17065 raft_consensus.cc:3032] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:49.852669 14627 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:35:49.940305 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 155125us
I20260806 20:35:49.981561 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.002435 14625 raft_consensus.cc:1226] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
W20260806 20:35:49.985343 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 225267us
I20260806 20:35:49.874706 14476 raft_consensus.cc:1261] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:35:49.877449 15153 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:50.004524 14478 raft_consensus.cc:2396] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:35:49.908548 17166 raft_consensus.cc:686] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:50.012799 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 135481us
W20260806 20:35:49.959142 17159 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.377s user 0.000s sys 0.001s
I20260806 20:35:49.956312 17137 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:49.954464 17065 raft_consensus.cc:729] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:50.020908 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.024204 14476 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:50.024801 16820 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.000s sys 0.003s
W20260806 20:35:50.025229 16820 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.528s user 0.000s sys 0.003s
I20260806 20:35:50.027549 16820 consensus_queue.cc:1035] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260806 20:35:50.021721 14473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.010s sys 0.014s
W20260806 20:35:50.000813 16962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.000s sys 0.012s
W20260806 20:35:50.030493 16962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.001s sys 0.013s
I20260806 20:35:50.030891 16962 consensus_queue.cc:1225] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer 92828dd6dbe74f29a04d4e702b502621 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260806 20:35:50.031678 14473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.010s sys 0.014s
I20260806 20:35:49.965782 17175 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:50.016896 17002 consensus_meta.cc:287] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.537s user 0.019s sys 0.000s
I20260806 20:35:50.041208 14469 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:50.037770 17046 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:50.046173 17168 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20260806 20:35:49.961699 17103 consensus_queue.cc:1046] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260806 20:35:50.032945 17180 raft_consensus.cc:2781] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:50.057957 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:49.931594 17181 consensus_queue.cc:1035] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260806 20:35:50.059280 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.068176 14551 raft_consensus.cc:2396] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:35:50.063481 14469 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:50.064656 14595 connection.cc:232] RPC call timeout handler was delayed by -4.0462105780025013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23740000000000003s for a response
W20260806 20:35:50.070283 14595 connection.cc:232] RPC call timeout handler was delayed by -3.63491057799547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2831s for a response
I20260806 20:35:50.072014 14625 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260806 20:35:50.087627 14595 connection.cc:232] RPC call timeout handler was delayed by -3.0832951549964491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
W20260806 20:35:50.088155 14595 connection.cc:232] RPC call timeout handler was delayed by -2.9509951549989637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17070000000000002s for a response
W20260806 20:35:50.088522 14595 connection.cc:232] RPC call timeout handler was delayed by -2.724195154994959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19590000000000002s for a response
W20260806 20:35:50.088866 14595 connection.cc:232] RPC call timeout handler was delayed by -2.724195154994959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19590000000000002s for a response
W20260806 20:35:50.089208 14595 connection.cc:232] RPC call timeout handler was delayed by -1.6477951550041325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3155s for a response
I20260806 20:35:50.092636 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.095016 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:50.096050 14625 raft_consensus.cc:2379] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:50.097231 17163 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.365s user 0.000s sys 0.001s
I20260806 20:35:50.097779 17187 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:50.078537 17119 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:50.098438 17187 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:50.099086 17187 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:50.082319 17191 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
W20260806 20:35:50.102131 17179 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260806 20:35:50.108413 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.108834 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1828) took 1345 ms. Trace:
I20260806 20:35:50.109236 14548 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 2.
W20260806 20:35:50.112003 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 49740.9us
I20260806 20:35:50.112382 17162 raft_consensus.cc:686] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:50.114559 15150 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:50.114432 17162 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.117090 15150 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:50.118777 15150 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:50.120498 14629 consensus_meta.cc:287] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.625s user 0.012s sys 0.000s
I20260806 20:35:50.120846 14629 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:50.109378 14633 rpcz_store.cc:276] 0806 20:35:48.764764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.820241 (+1055477us) service_pool.cc:227] Handling call
0806 20:35:49.945836 (+125595us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:49.956685 (+ 10849us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:35:50.000388 (+ 43703us) log.cc:842] Serialized 14 byte log entry
0806 20:35:50.107083 (+106695us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:50.107146 (+ 63us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:50.107200 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:50.107281 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:50.107419 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":55242,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"thread_start_us":55481,"threads_started":1}
I20260806 20:35:50.133144 14441 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:50.137068 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.036552 17137 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:50.044832 17002 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.093456 16974 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:50.146517 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 422786us
I20260806 20:35:50.147794 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.148741 14551 raft_consensus.cc:2396] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:35:50.148494 14518 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:50.150261 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.103273 17180 raft_consensus.cc:686] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:50.157528 14518 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:50.158479 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:50.160214 14442 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:50.161371 14632 raft_consensus.cc:2379] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:50.162329 14518 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:50.103679 16844 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260806 20:35:50.147421 16820 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:35:50.177242 14518 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:50.178603 14629 raft_consensus.cc:1226] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 20:35:50.233219 14550 raft_consensus.cc:1261] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:35:50.233016 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.233073 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.253281 14555 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:50.254230 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.254343 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.255421 14548 raft_consensus.cc:2379] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:50.175477 17158 consensus_meta.cc:287] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.845s user 0.015s sys 0.000s
I20260806 20:35:50.237372 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:50.234347 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 57129.7us
I20260806 20:35:50.241123 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.258412 14596 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:50.243597 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.168972 17110 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:50.161584 17197 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:50.257532 14473 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:50.260612 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.257762 14518 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:50.264006 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.264338 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.189628 17181 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:35:50.176383 17182 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:50.280304 14518 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:50.280442 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:50.281306 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.277439 17197 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:50.283257 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.286202 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.282928 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.295842 14546 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:50.296335 14546 raft_consensus.cc:2379] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 2: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:50.240281 16962 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:50.159448 16974 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.293341 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:50.297098 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.299840 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:50.261209 17176 log.cc:924] Time spent T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.322s user 0.001s sys 0.001s
I20260806 20:35:50.300647 14552 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:50.315606 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.316743 15187 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:50.317878 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 153616us
I20260806 20:35:50.279687 17059 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:50.328641 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:50.321527 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 161388us
I20260806 20:35:50.238543 16820 consensus_queue.cc:1046] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260806 20:35:50.331602 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.187812 17116 consensus_queue.cc:1035] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260806 20:35:50.286175 17181 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:50.328091 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.329285 14473 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:50.348402 14596 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:50.159015 17166 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.351567 14550 raft_consensus.cc:2379] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:50.377534 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.260583 17158 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.364445 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.296940 17197 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:50.328778 17059 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:50.396051 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 102798us
I20260806 20:35:50.331045 17010 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:35:50.428189 14442 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:50.367411 17138 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260806 20:35:50.359515 17181 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:35:50.467593 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.430137 14442 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:50.381943 17110 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:50.468688 15187 raft_consensus.cc:2379] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 8: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:50.381611 16942 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:50.465281 15183 raft_consensus.cc:2379] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:50.364506 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.366019 17186 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:50.474797 17158 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:50.475081 16947 consensus_meta.cc:287] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.043s user 0.011s sys 0.001s
I20260806 20:35:50.475595 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:50.391319 17193 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:50.302495 17182 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.383455 17194 consensus_queue.cc:1046] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.370s
I20260806 20:35:50.380123 17180 consensus_queue.cc:237] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.559175 16820 consensus_queue.cc:1225] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer 148bf2a11dba423ab364371f0458e871 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 20:35:50.657495 17209 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:50.508143 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.675715 14552 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:50.477001 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:50.557540 14521 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.010s sys 0.039s
W20260806 20:35:50.699443 14521 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.761s user 0.019s sys 0.051s
W20260806 20:35:50.699869 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 889616us
I20260806 20:35:50.425765 17059 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:50.673700 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1566) took 1374 ms. Trace:
W20260806 20:35:50.682874 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 252869us
I20260806 20:35:50.629498 15184 raft_consensus.cc:1261] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:35:50.676199 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.658234 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.721347 14478 raft_consensus.cc:2379] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:50.720137 14553 rpcz_store.cc:276] 0806 20:35:49.299099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.362449 (+ 63350us) service_pool.cc:227] Handling call
0806 20:35:49.373702 (+ 11253us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:49.448622 (+ 74920us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:49.448750 (+ 128us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:49.481854 (+ 33104us) log.cc:842] Serialized 26 byte log entry
0806 20:35:49.720568 (+238714us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:49.720620 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:49.720715 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:50.673453 (+952738us) raft_consensus.cc:1671] finished
0806 20:35:50.673476 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:50.673637 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":203808,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":19097,"thread_start_us":203607,"threads_started":2,"wal-append.queue_time_us":241977}
I20260806 20:35:50.721905 14476 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:50.722491 14476 raft_consensus.cc:2396] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:35:50.508750 14595 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:50.720536 14442 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:50.462309 17002 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:50.529173 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.589674 17203 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:50.673231 17206 log.cc:924] Time spent T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.197s user 0.000s sys 0.000s
I20260806 20:35:50.480628 17103 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:50.781150 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:50.769091 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:50.781875 14474 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:50.769308 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:50.769374 17211 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:50.719445 16947 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:50.770079 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 49685.1us
I20260806 20:35:50.719689 16820 raft_consensus.cc:3032] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:50.771420 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:50.773281 17201 log.cc:924] Time spent T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.341s user 0.000s sys 0.000s
W20260806 20:35:50.774243 17215 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260806 20:35:50.797360 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:50.768689 17207 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.410s user 0.000s sys 0.001s
I20260806 20:35:50.476682 17110 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:50.676071 17193 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:50.556830 17205 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:50.782748 14553 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:50.673346 17137 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:50.775270 17203 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:50.811290 17218 log.cc:924] Time spent T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20260806 20:35:50.798542 17211 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:50.878633 17211 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:50.783066 16942 consensus_queue.cc:1046] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 20:35:50.812071 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1569) took 1319 ms. Trace:
I20260806 20:35:50.672480 17143 consensus_queue.cc:260] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:50.882728 17227 raft_consensus.cc:3032] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:50.883160 17227 raft_consensus.cc:729] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:50.812999 17202 log.cc:924] Time spent T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.514s user 0.001s sys 0.000s
I20260806 20:35:50.884899 17224 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:50.885548 17224 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:50.885990 17224 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:50.774170 17194 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:50.775336 17002 raft_consensus.cc:2741] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260806 20:35:50.833999 17154 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 1.457s user 0.005s sys 0.000s
I20260806 20:35:50.834270 16947 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:50.892663 17219 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:50.789736 17103 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:50.893661 17103 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:50.836391 14474 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260806 20:35:50.850952 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.031s sys 0.061s
I20260806 20:35:50.865736 14596 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:50.866901 17196 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:50.870059 17198 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.344s user 0.000s sys 0.001s
I20260806 20:35:50.799465 16820 raft_consensus.cc:729] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:50.880579 14547 rpcz_store.cc:276] 0806 20:35:49.492484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.003934 (+511450us) service_pool.cc:227] Handling call
0806 20:35:50.035330 (+ 31396us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:50.035640 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:50.035677 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:50.040963 (+ 5286us) log.cc:842] Serialized 26 byte log entry
0806 20:35:50.157602 (+116639us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:50.157686 (+ 84us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:50.157885 (+ 199us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:50.811855 (+653970us) raft_consensus.cc:1671] finished
0806 20:35:50.811874 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:50.812010 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":92970,"threads_started":1,"wal-append.queue_time_us":321790}
W20260806 20:35:51.181689 14546 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.788s user 0.019s sys 0.005s
W20260806 20:35:50.834849 17216 log.cc:924] Time spent T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.176s user 0.000s sys 0.000s
I20260806 20:35:50.591499 17190 consensus_queue.cc:1035] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.580s
I20260806 20:35:50.929241 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:50.858575 17110 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:50.780799 17209 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:50.894294 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:51.011848 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:51.085302 14444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.002s sys 0.011s
I20260806 20:35:50.863037 17193 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:51.179811 14552 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:50.947768 14553 raft_consensus.cc:1261] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:35:50.931632 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:51.182325 14549 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.011s sys 0.014s
I20260806 20:35:51.250424 15183 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:51.250684 14549 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.823s user 0.012s sys 0.014s
W20260806 20:35:50.939038 17207 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260806 20:35:50.884099 17227 consensus_queue.cc:260] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:51.211292 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 746376us
I20260806 20:35:50.872884 17205 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:51.177493 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:50.838064 17010 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:51.177922 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1573) took 1506 ms. Trace:
W20260806 20:35:51.182271 17183 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.017s sys 0.007s
I20260806 20:35:50.893373 17219 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:51.185741 14441 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.043s user 0.015s sys 0.010s
W20260806 20:35:51.246553 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 234219us
W20260806 20:35:51.225397 16962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.858s user 0.004s sys 0.026s
W20260806 20:35:51.084398 17153 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.801s user 0.011s sys 0.040s
I20260806 20:35:51.284201 14521 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:51.247581 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 381963us
I20260806 20:35:50.886955 17203 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:51.215854 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.096s user 0.042s sys 0.071s
W20260806 20:35:51.216997 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.005s sys 0.023s
W20260806 20:35:51.287487 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.972s user 0.018s sys 0.039s
W20260806 20:35:51.287835 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 1.00757e+06us
I20260806 20:35:51.250273 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1700) took 1065 ms. Trace:
I20260806 20:35:51.288981 14553 rpcz_store.cc:276] 0806 20:35:50.184657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.876391 (+691734us) service_pool.cc:227] Handling call
0806 20:35:50.947331 (+ 70940us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:51.249983 (+302652us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:51.250209 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:51.210270 14546 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.826s user 0.021s sys 0.006s
W20260806 20:35:51.070981 17112 consensus_meta.cc:287] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.340s user 0.010s sys 0.018s
I20260806 20:35:51.251346 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1575) took 1104 ms. Trace:
W20260806 20:35:51.254707 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 242989us
W20260806 20:35:51.255491 14628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1847) took 1950 ms (client timeout 1609 ms). Trace:
I20260806 20:35:50.937517 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:50.934557 17208 log.cc:924] Time spent T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.352s user 0.000s sys 0.002s
I20260806 20:35:50.922995 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:50.936305 17204 log.cc:924] Time spent T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.671s user 0.001s sys 0.000s
W20260806 20:35:51.277287 17183 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.019s sys 0.008s
W20260806 20:35:51.274822 14441 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.132s user 0.015s sys 0.013s
W20260806 20:35:51.304334 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 1.16142e+06us
W20260806 20:35:51.246373 14444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.002s sys 0.020s
W20260806 20:35:51.286274 17153 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s user 0.029s sys 0.051s
I20260806 20:35:51.272683 17205 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.271330 17015 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:51.215837 17182 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:51.281379 17010 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.316989 17010 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:51.319749 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:51.281229 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:51.271327 14477 rpcz_store.cc:276] 0806 20:35:49.671161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.671340 (+ 179us) service_pool.cc:227] Handling call
0806 20:35:49.676992 (+ 5652us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:49.844054 (+167062us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:49.844096 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:49.959137 (+115041us) log.cc:842] Serialized 52 byte log entry
0806 20:35:50.056551 (+ 97414us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:50.102420 (+ 45869us) log.cc:842] Serialized 14 byte log entry
0806 20:35:50.102801 (+ 381us) log.cc:842] Serialized 14 byte log entry
0806 20:35:50.102962 (+ 161us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:50.103040 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:51.177696 (+1074656us) raft_consensus.cc:1671] finished
0806 20:35:51.177717 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:51.177858 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19134,"prepare.queue_time_us":46624,"prepare.run_wall_time_us":64,"raft.queue_time_us":26307,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":38413,"spinlock_wait_cycles":11526400,"thread_start_us":181570,"threads_started":2,"wal-append.queue_time_us":198841}
I20260806 20:35:51.320405 14633 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.290356 14596 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:51.286931 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 1.16816e+06us
I20260806 20:35:51.321308 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:51.321935 15183 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:51.322851 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:51.324177 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:51.219878 16974 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:51.324694 15183 raft_consensus.cc:2362] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7. Re-sending same reply.
I20260806 20:35:51.297369 17112 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.325829 16974 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:51.326764 16974 raft_consensus.cc:686] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:51.327189 17112 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:51.328001 14629 raft_consensus.cc:1226] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:51.327764 16974 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.313117 14441 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:51.293123 14628 rpcz_store.cc:269] 0806 20:35:49.305061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.321098 (+ 16037us) service_pool.cc:227] Handling call
0806 20:35:50.894143 (+1573045us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593bce 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:51.255376 (+361233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11371776}
I20260806 20:35:51.334074 14441 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.298226 14442 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.335253 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:50.939754 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:51.300249 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:51.300647 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:51.297503 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 413) took 1015 ms. Trace:
I20260806 20:35:51.375084 15186 rpcz_store.cc:276] 0806 20:35:50.281929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.303965 (+ 22036us) service_pool.cc:227] Handling call
0806 20:35:50.922786 (+618821us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593b7c 00007fb9b059373b 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:51.297428 (+374642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11334400}
I20260806 20:35:51.303006 14595 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:51.276998 15187 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:35:51.275485 17219 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:51.377175 17137 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 20:35:51.377314 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 85483.8us
I20260806 20:35:51.377707 17137 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:51.378211 17137 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:51.378371 14595 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Timed out: Call waited in the queue past client deadline [suppressed 40 similar messages]
I20260806 20:35:51.307097 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:51.380012 14628 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
W20260806 20:35:51.277673 16962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.923s user 0.004s sys 0.044s
I20260806 20:35:51.292093 16820 consensus_queue.cc:260] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.342334 14633 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:51.293459 14549 rpcz_store.cc:276] 0806 20:35:50.147213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.147365 (+ 152us) service_pool.cc:227] Handling call
0806 20:35:50.165816 (+ 18451us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:50.166052 (+ 236us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:50.287016 (+120964us) log.cc:842] Serialized 14 byte log entry
0806 20:35:51.250984 (+963968us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:51.251033 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:51.251087 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:51.251161 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:51.251288 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":264174,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"thread_start_us":1051479,"threads_started":3,"wal-append.queue_time_us":304464}
W20260806 20:35:51.309033 16910 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.546s user 0.020s sys 0.019s
I20260806 20:35:51.291821 17237 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:51.412755 14553 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:51.308005 15181 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 3.
W20260806 20:35:51.422451 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 44141.6us
W20260806 20:35:51.409970 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 109852us
I20260806 20:35:51.313817 17015 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:51.401077 16962 raft_consensus.cc:686] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:51.251461 17196 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.431986 14518 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.439122 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:51.447068 14595 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:51.435081 17207 log.cc:924] Time spent T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260806 20:35:51.449378 16820 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.443841 15180 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
W20260806 20:35:51.476403 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 48342.3us
W20260806 20:35:51.445398 16854 consensus_meta.cc:287] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.723s user 0.011s sys 0.012s
W20260806 20:35:51.445946 17142 consensus_meta.cc:287] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.776s user 0.011s sys 0.014s
I20260806 20:35:51.438007 17015 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.417884 16910 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.389766 17010 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
W20260806 20:35:51.449553 16935 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.628s user 0.008s sys 0.011s
W20260806 20:35:51.454509 17140 consensus_meta.cc:287] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.677s user 0.000s sys 0.023s
W20260806 20:35:51.454720 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 170620us
I20260806 20:35:51.495229 14521 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:51.474119 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 140125us
I20260806 20:35:51.478621 14518 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:51.447224 17239 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:51.426162 17230 consensus_queue.cc:1035] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
W20260806 20:35:51.500308 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 202295us
I20260806 20:35:51.501092 14442 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:51.478509 17181 consensus_meta.cc:287] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.972s user 0.018s sys 0.004s
W20260806 20:35:51.488212 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 49258.2us
I20260806 20:35:51.485759 17189 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:51.497455 14441 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.473891 17240 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:51.483248 17229 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:51.504153 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1572) took 1872 ms. Trace:
I20260806 20:35:51.423877 17237 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.508621 14470 rpcz_store.cc:276] 0806 20:35:49.632266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.665813 (+ 33547us) service_pool.cc:227] Handling call
0806 20:35:49.688358 (+ 22545us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:50.019707 (+331349us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:50.046802 (+ 27095us) log.cc:842] Serialized 14 byte log entry
0806 20:35:50.141664 (+ 94862us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:50.144486 (+ 2822us) log.cc:842] Serialized 26 byte log entry
0806 20:35:50.144751 (+ 265us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:50.146031 (+ 1280us) log.cc:842] Serialized 14 byte log entry
0806 20:35:50.149218 (+ 3187us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:50.149303 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:51.503942 (+1354639us) raft_consensus.cc:1671] finished
0806 20:35:51.503966 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:51.504087 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2878,"prepare.queue_time_us":9118,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":270640,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":26117,"spinlock_wait_cycles":5600512,"thread_start_us":384089,"threads_started":2,"wal-append.queue_time_us":215015}
I20260806 20:35:51.503631 17239 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:51.482167 16854 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.526180 14518 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:51.484997 17142 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.500185 17209 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:51.531463 17254 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:51.531723 14442 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:51.538898 15150 leader_election.cc:334] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789): Timed out: RequestConsensusVote RPC to 127.12.248.129:44789 timed out after 2.163s (SENT)
I20260806 20:35:51.484350 17193 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.510694 17240 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:51.510957 17229 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:51.496696 16935 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.499248 17140 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:51.549760 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 52363.9us
I20260806 20:35:51.553570 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:51.544982 16962 consensus_queue.cc:237] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.556766 17244 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:51.556110 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:51.561497 14441 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.521683 17239 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.557215 17244 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:51.576030 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:51.577523 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 143327us
I20260806 20:35:51.545850 17249 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:51.577565 14595 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:51.550251 17229 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:51.510740 17181 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:51.581131 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 85984.6us
I20260806 20:35:51.601639 14628 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:35:51.628410 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:51.550086 17240 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:51.735023 14628 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:35:51.574029 17010 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:35:51.749994 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 188588us
I20260806 20:35:51.750905 14441 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:51.626149 17211 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.747s user 0.013s sys 0.003s
W20260806 20:35:51.753579 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 221924us
W20260806 20:35:51.634565 17247 log.cc:924] Time spent T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260806 20:35:51.580442 17190 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:51.756820 14521 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.585830 16974 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:51.575312 17244 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.700742 17249 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:51.599241 17254 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:51.754547 14442 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.633438 17233 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:51.633672 17241 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:51.633267 17256 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:35:51.763474 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 189870us
I20260806 20:35:51.572899 17189 consensus_queue.cc:1035] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.183s
I20260806 20:35:51.598798 17112 consensus_queue.cc:1035] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.174s
W20260806 20:35:51.769753 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 460799us
I20260806 20:35:51.755736 17190 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:51.758610 17196 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:51.775621 16910 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:51.775885 17140 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:51.779819 17189 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:51.783433 17260 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:51.765506 17249 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:51.632403 17232 consensus_queue.cc:1035] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.196s
I20260806 20:35:51.789712 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:51.838315 14628 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:51.766955 17233 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:51.767643 16935 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:51.752718 17203 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.774614 16854 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:51.793144 17193 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:51.755868 17211 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:51.825522 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 71083.2us
W20260806 20:35:51.828109 17252 log.cc:924] Time spent T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
W20260806 20:35:51.830842 17175 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.760s user 0.017s sys 0.012s
I20260806 20:35:51.778086 17254 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:35:51.832479 17170 consensus_meta.cc:287] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.957s user 0.028s sys 0.021s
I20260806 20:35:51.808699 17190 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.815237 17256 raft_consensus.cc:2741] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:35:51.823328 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:51.844672 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 65439.7us
I20260806 20:35:51.764783 17258 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
I20260806 20:35:51.849165 14442 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.768083 17183 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260806 20:35:51.853298 14551 raft_consensus.cc:2379] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:51.860527 14633 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.843722 17233 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:51.843698 17010 raft_consensus.cc:686] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:51.847291 16935 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:51.860925 14444 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:51.874418 17203 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:51.847396 17260 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:51.935102 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:51.949509 14630 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:51.859613 17175 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.861406 17170 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.921691 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:51.940083 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:51.930132 17187 consensus_meta.cc:287] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.830s user 0.015s sys 0.035s
I20260806 20:35:51.934213 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:51.931926 17059 consensus_meta.cc:287] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.164s user 0.010s sys 0.015s
W20260806 20:35:51.929090 17252 log.cc:924] Time spent T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260806 20:35:51.927434 17205 consensus_meta.cc:287] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.556s user 0.000s sys 0.015s
I20260806 20:35:51.898226 17248 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:51.903808 17110 consensus_meta.cc:287] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.628s user 0.007s sys 0.018s
W20260806 20:35:51.900727 17224 consensus_meta.cc:287] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.014s user 0.007s sys 0.005s
I20260806 20:35:51.910168 17181 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:51.931735 14628 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:51.903963 17244 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.859879 17237 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:51.910956 17197 consensus_meta.cc:287] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.446s user 0.013s sys 0.012s
W20260806 20:35:51.907152 17219 consensus_meta.cc:287] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.530s user 0.006s sys 0.005s
I20260806 20:35:51.922997 17190 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.898792 17211 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:51.897138 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48059.3us
W20260806 20:35:51.897485 15151 leader_election.cc:334] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.967s (SENT)
I20260806 20:35:52.023442 15151 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:52.024719 14442 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:52.025820 17237 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:52.026324 17237 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:52.026672 17237 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:52.026602 17110 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.027174 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:51.902678 17233 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:51.907040 17249 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.915936 14633 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:35:51.921767 17137 consensus_meta.cc:287] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.543s user 0.009s sys 0.004s
I20260806 20:35:51.954967 17010 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:51.957008 16969 raft_consensus.cc:3032] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:35:51.959789 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 382290us
I20260806 20:35:52.036175 14595 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:51.968616 16935 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:51.961740 15185 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:51.962947 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 102103us
I20260806 20:35:51.926486 17229 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.022680 14555 raft_consensus.cc:2354] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
W20260806 20:35:52.030190 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 279346us
I20260806 20:35:52.030186 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:52.044150 14633 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:35:52.031257 17110 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:52.034837 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.032275 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:51.960089 14630 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
W20260806 20:35:52.039305 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 141870us
I20260806 20:35:52.040632 14444 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:52.042325 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:51.967862 17059 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.046809 14628 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:35:52.038991 15182 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 470) took 1848 ms (client timeout 2223 ms). Trace:
I20260806 20:35:52.050746 16820 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:35:52.050676 15182 rpcz_store.cc:269] 0806 20:35:50.189285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.256862 (+ 67577us) service_pool.cc:227] Handling call
0806 20:35:51.934096 (+1677234us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593c6c 00007fb9b0593bd6 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:52.037761 (+103665us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89153920}
I20260806 20:35:52.037294 16969 raft_consensus.cc:729] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:52.037842 17137 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.052292 16969 consensus_queue.cc:260] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.053089 17249 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:52.053634 16969 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:52.054008 17170 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.057052 17233 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:52.057376 17121 log.cc:924] Time spent T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 2.630s user 0.004s sys 0.000s
I20260806 20:35:52.058876 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 356) took 5723 ms. Trace:
I20260806 20:35:52.059216 14554 rpcz_store.cc:276] 0806 20:35:46.335005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:47.857864 (+1522859us) service_pool.cc:227] Handling call
0806 20:35:47.858258 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:47.884125 (+ 25867us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:47.884165 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:47.884739 (+ 574us) log.cc:842] Serialized 26 byte log entry
0806 20:35:48.630342 (+745603us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:48.630383 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:48.630454 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:52.056777 (+3426323us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:52.058561 (+ 1784us) raft_consensus.cc:1671] finished
0806 20:35:52.058584 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:52.058797 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":331,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"spinlock_wait_cycles":7489920,"thread_start_us":745058,"threads_started":1,"wal-append.queue_time_us":1411249}
I20260806 20:35:52.061127 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:52.061333 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1845) took 2813 ms (client timeout 1753 ms). Trace:
I20260806 20:35:52.061399 17244 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.037401 17183 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:35:52.023036 17239 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.038177 17211 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:35:52.051519 17103 consensus_meta.cc:287] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.129s user 0.023s sys 0.027s
I20260806 20:35:52.063376 14554 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:52.063625 14630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1860) took 2365 ms (client timeout 1910 ms). Trace:
I20260806 20:35:52.047667 14441 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:52.061791 14626 rpcz_store.cc:269] 0806 20:35:49.247843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.321204 (+ 73361us) service_pool.cc:227] Handling call
0806 20:35:52.061119 (+2739915us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003bf954. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:52.061245 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2318402176}
I20260806 20:35:52.048776 14442 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:52.072141 14630 rpcz_store.cc:269] 0806 20:35:49.681908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.739237 (+ 57329us) service_pool.cc:227] Handling call
0806 20:35:51.934933 (+2195696us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:52.047385 (+112452us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5449984}
I20260806 20:35:51.965806 17187 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.070546 17275 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260806 20:35:52.062585 17175 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:52.079185 17015 consensus_meta.cc:287] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.585s user 0.014s sys 0.010s
I20260806 20:35:52.046681 17274 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:52.023447 17224 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.080833 17229 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.022810 17219 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.126776 14547 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: []
I20260806 20:35:52.126984 14628 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:52.084491 17190 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:52.085259 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:52.071038 14595 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:52.087072 14444 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:52.098464 14633 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 9.
I20260806 20:35:52.042647 16935 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:52.051432 16820 raft_consensus.cc:686] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:52.101646 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:51.977101 17205 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.100229 15185 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
W20260806 20:35:52.105664 17178 log.cc:924] Time spent T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 2.126s user 0.002s sys 0.000s
I20260806 20:35:52.105957 17239 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.106379 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:52.099771 17110 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:35:52.110190 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 62578.6us
I20260806 20:35:52.140750 16935 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:52.112208 14554 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:52.024941 17197 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:52.083249 17149 log.cc:924] Time spent T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 2.589s user 0.004s sys 0.000s
I20260806 20:35:52.131520 14631 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:35:52.137540 14628 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
W20260806 20:35:52.140615 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 53630.9us
W20260806 20:35:52.142222 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 71286.5us
W20260806 20:35:52.173262 14631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1859) took 2491 ms (client timeout 1865 ms). Trace:
W20260806 20:35:52.143095 15185 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 414) took 1841 ms (client timeout 1879 ms). Trace:
I20260806 20:35:52.121898 17260 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:52.173612 14444 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:52.075273 17240 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.173691 14595 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:52.173667 14631 rpcz_store.cc:269] 0806 20:35:49.681525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.738823 (+ 57298us) service_pool.cc:227] Handling call
0806 20:35:52.085101 (+2346278us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:52.173188 (+ 88087us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6058368}
I20260806 20:35:52.174557 14444 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:52.175285 14595 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:52.175666 17240 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:52.109241 17065 consensus_queue.cc:260] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.075470 17103 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.132350 17155 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:35:52.111326 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 62663.1us
I20260806 20:35:52.158670 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.161161 17256 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:52.159066 14596 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:52.161528 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:52.162375 17254 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:52.163691 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:52.164904 14554 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:52.165563 17219 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:52.166095 17275 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:52.122119 17015 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.167063 17059 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:52.107124 17237 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.153033 17110 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:52.144982 17274 raft_consensus.cc:686] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:52.170528 17205 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:52.148437 17265 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:52.171649 15180 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:35:52.176406 14595 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:52.178761 14548 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
W20260806 20:35:52.180124 15185 rpcz_store.cc:269] 0806 20:35:50.301683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.316142 (+ 14459us) service_pool.cc:227] Handling call
0806 20:35:51.939884 (+1623742us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:52.143034 (+203150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25221888}
I20260806 20:35:52.209298 15183 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:52.211030 14442 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:52.238636 14442 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:52.240187 14442 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:52.193074 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:52.213074 14553 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 8: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:52.213927 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 54993.1us
W20260806 20:35:52.242627 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 66297.3us
I20260806 20:35:52.243883 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:52.245088 14595 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:52.246270 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:52.176793 17232 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:52.176644 17190 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:52.144399 17263 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:52.181133 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:52.179577 17239 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:52.221467 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:52.193063 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:52.251251 15184 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:52.251531 14550 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260806 20:35:52.218262 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 55006.7us
I20260806 20:35:52.177515 17224 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.254571 14595 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:52.178486 17260 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:52.173918 17197 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.199899 17065 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:52.176719 17137 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:52.256440 14630 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:52.252795 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1493) took 2811 ms. Trace:
I20260806 20:35:52.177649 17266 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:52.276835 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:52.231016 17010 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:52.280412 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:52.248684 17110 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:52.281031 15183 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:52.224962 15181 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:52.249611 17232 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:52.283990 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.250639 17196 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:52.250485 17155 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:52.250277 17219 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:35:52.257195 16974 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:52.176555 16820 consensus_queue.cc:237] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.278698 17059 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:52.295418 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.296025 14553 raft_consensus.cc:2379] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:52.257521 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:52.286109 17232 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:52.286648 17190 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:35:52.259184 17112 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260806 20:35:52.305989 17015 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:52.307448 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1756) took 3116 ms. Trace:
I20260806 20:35:52.308900 17285 consensus_queue.cc:1035] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20260806 20:35:52.311795 17279 consensus_queue.cc:1046] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
W20260806 20:35:52.312609 17172 log.cc:924] Time spent T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 2.554s user 0.003s sys 0.000s
I20260806 20:35:52.313148 17270 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:52.315393 17274 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.317663 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1689) took 2134 ms. Trace:
I20260806 20:35:52.353353 14548 rpcz_store.cc:276] 0806 20:35:50.182965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.302974 (+120009us) service_pool.cc:227] Handling call
0806 20:35:50.303346 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:52.251352 (+1948006us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:52.317363 (+ 66011us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54000b2cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:52.317578 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4312717696}
W20260806 20:35:52.319936 15187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 416) took 1964 ms (client timeout 1587 ms). Trace:
I20260806 20:35:52.247241 17275 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.290592 17219 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:52.224881 17256 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.257563 17239 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.296530 17273 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:52.213718 17254 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:52.354588 15187 rpcz_store.cc:269] 0806 20:35:50.355416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.483959 (+128543us) service_pool.cc:227] Handling call
0806 20:35:52.319701 (+1835742us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54003be2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:52.319841 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1225700224}
W20260806 20:35:52.360236 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 104448us
I20260806 20:35:52.337885 14471 rpcz_store.cc:276] 0806 20:35:49.190588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.303326 (+112738us) service_pool.cc:227] Handling call
0806 20:35:49.361959 (+ 58633us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:49.571279 (+209320us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:49.571325 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:49.583180 (+ 11855us) log.cc:842] Serialized 26 byte log entry
0806 20:35:49.754523 (+171343us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:49.799215 (+ 44692us) log.cc:842] Serialized 14 byte log entry
0806 20:35:51.845308 (+2046093us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b180045bce0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05c8155 00007fb9b34eb3d0 00007fb9b34fd576 00007fb9b350e0d7 00007fb9b350e058
0806 20:35:52.043546 (+198238us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:52.104907 (+ 61361us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:35:52.307090 (+202183us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b54003c4e54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:52.307145 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:52.307212 (+ 67us) raft_consensus.cc:1671] finished
0806 20:35:52.307228 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:52.307382 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10853,"prepare.queue_time_us":44227,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":41,"raft.queue_time_us":226680,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":213957632,"thread_start_us":335434,"threads_started":2,"wal-append.queue_time_us":206198}
I20260806 20:35:52.337558 17187 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.338640 17285 consensus_queue.cc:1225] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer 8e0cfd35f2994f91aaae46c25c6d6bff log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260806 20:35:52.374378 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 123620us
I20260806 20:35:52.381779 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.359149 17219 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:35:52.390429 14553 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:35:52.391615 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.392866 17103 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.401983 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.406108 17237 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:52.406446 14553 raft_consensus.cc:2379] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:52.406103 17254 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.407618 17046 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:52.408120 17046 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:52.408408 17275 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.409286 17275 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:52.409968 17275 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.410753 17232 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.411038 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.365870 17270 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260806 20:35:52.412925 17289 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:52.413481 17289 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.413883 17143 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:52.409216 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1701) took 2224 ms. Trace:
I20260806 20:35:52.412223 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.416425 14547 rpcz_store.cc:276] 0806 20:35:50.184799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:51.213500 (+1028701us) service_pool.cc:227] Handling call
0806 20:35:51.270385 (+ 56885us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:52.127252 (+856867us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:52.408924 (+281672us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54000b40d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:52.409147 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1969101568}
W20260806 20:35:52.421736 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 137866us
I20260806 20:35:52.425879 15183 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:52.380599 17263 raft_consensus.cc:686] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:52.429952 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1474) took 3973 ms (client timeout 2775 ms). Trace:
W20260806 20:35:52.430409 14627 rpcz_store.cc:269] 0806 20:35:48.456627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:48.499820 (+ 43193us) service_pool.cc:227] Handling call
0806 20:35:52.429784 (+3929964us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002be3d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:52.429881 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3108405120}
I20260806 20:35:52.439081 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:52.439857 15183 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:52.441506 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:52.442481 14547 raft_consensus.cc:2379] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:52.445550 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.396591 17266 raft_consensus.cc:686] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:52.447846 14441 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:52.448838 14478 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260806 20:35:52.448330 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 167959us
I20260806 20:35:52.452670 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.453505 14479 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:35:52.454344 14473 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 20:35:52.464671 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:52.471901 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.472599 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.473682 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:52.474167 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.474699 14478 raft_consensus.cc:2354] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:52.474782 14553 raft_consensus.cc:3037] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:52.476445 14474 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 20:35:52.473557 17275 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.441339 17289 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.477077 15186 raft_consensus.cc:1226] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
W20260806 20:35:52.638130 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 190364us
I20260806 20:35:52.639465 14441 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:52.455960 17279 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:52.566246 14548 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:52.610320 15183 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:52.565785 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:52.647516 14547 raft_consensus.cc:2396] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:35:52.551703 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:52.543527 14479 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 9.
I20260806 20:35:52.647325 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.649478 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.649693 14476 raft_consensus.cc:2354] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:52.650559 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.477643 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.651332 14476 raft_consensus.cc:2354] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:52.653041 15183 raft_consensus.cc:3032] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:52.655012 15183 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:52.656062 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:52.506021 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:52.656800 14476 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:52.657598 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:52.658190 14476 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:52.659094 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:52.659723 14476 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:52.657720 15183 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.468308 17232 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:52.477448 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:52.477774 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:52.664278 17279 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.639770 15178 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:52.696811 15183 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:52.668411 17266 consensus_queue.cc:237] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.468916 17165 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.231s
I20260806 20:35:52.469416 17183 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.235s
I20260806 20:35:52.545878 17272 consensus_queue.cc:1046] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.198s
I20260806 20:35:52.660564 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.480216 17263 consensus_queue.cc:237] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.277848 14625 rpcz_store.cc:276] 0806 20:35:49.441224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.130843 (+689619us) service_pool.cc:227] Handling call
0806 20:35:50.131206 (+ 363us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:50.144474 (+ 13268us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:35:50.163912 (+ 19438us) log.cc:842] Serialized 14 byte log entry
0806 20:35:52.201199 (+2037287us) spinlock_profiling.cc:243] Waited 7.5 ms on lock 0x7b180040b580. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05424ca 00007fb9b9c23f99 00007fb9b739bbcf 00007fb9b05cb7b6 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d
0806 20:35:52.202781 (+ 1582us) spinlock_profiling.cc:243] Waited 9.97 ms on lock 0x7b180040b580. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05ad345 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a57b 00007fb9b34fd88a
0806 20:35:52.252481 (+ 49700us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:52.252534 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:52.252591 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:52.252648 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:52.252747 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53327,"prepare.queue_time_us":19246,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":1974,"raft.queue_time_us":80,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":5710,"spinlock_wait_cycles":49545600,"thread_start_us":32551,"threads_started":1,"wal-append.queue_time_us":2084501}
I20260806 20:35:52.716580 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:52.753937 15182 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:52.754339 14469 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:52.857223 17113 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260806 20:35:52.881830 17235 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:52.745790 17191 raft_consensus.cc:2781] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:52.870339 17142 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.882349 17235 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:52.875733 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.883833 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.538461 16935 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.730084 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:52.893296 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.894127 14478 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:52.779269 14548 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:52.703337 14474 raft_consensus.cc:2362] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4. Re-sending same reply.
I20260806 20:35:52.861232 14476 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:52.887895 15150 leader_election.cc:398] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:52.555723 17275 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:52.860687 14553 raft_consensus.cc:2454] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:52.898082 15150 leader_election.cc:401] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260806 20:35:52.714111 15184 raft_consensus.cc:2454] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:52.885460 14479 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:52.900395 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:52.900990 14553 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 7
W20260806 20:35:52.845144 13287 debug-util.cc:398] Leaking SignalData structure 0x7b0800390ae0 after lost signal to thread 14518
W20260806 20:35:52.906366 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 485408us
I20260806 20:35:52.884793 17010 raft_consensus.cc:686] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:52.907256 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:52.909873 16935 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:52.911052 17279 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.912726 17303 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:52.913300 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:52.914556 17190 consensus_meta.cc:287] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.577s user 0.010s sys 0.002s
W20260806 20:35:52.915170 17110 consensus_meta.cc:287] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.625s user 0.016s sys 0.004s
I20260806 20:35:52.893534 16855 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:52.916543 14474 raft_consensus.cc:3037] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:52.750437 17219 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:52.895236 15178 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 471) took 2658 ms (client timeout 1642 ms). Trace:
I20260806 20:35:52.917066 17191 raft_consensus.cc:686] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:52.957713 15183 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:52.952647 14553 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
W20260806 20:35:52.966195 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 50682.6us
I20260806 20:35:52.910912 17059 consensus_queue.cc:1046] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.036s
I20260806 20:35:52.971434 14441 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:52.963661 14479 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:52.964525 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:52.973004 14470 raft_consensus.cc:1226] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260806 20:35:52.906509 13287 kernel_stack_watchdog.cc:198] Thread 14518 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/outbound_call.cc:308 for 304ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260806 20:35:52.947854 17295 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:52.970096 14476 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:35:52.975811 14547 raft_consensus.cc:2379] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:52.976289 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 73658.2us
I20260806 20:35:52.918442 17256 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:52.997516 15150 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:52.967190 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:52.999001 14474 raft_consensus.cc:2454] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:52.958887 17190 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:35:53.004557 17304 consensus_queue.cc:1035] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260806 20:35:53.000789 17260 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.008s sys 0.017s
W20260806 20:35:53.011037 17260 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.008s sys 0.018s
I20260806 20:35:53.012058 17260 raft_consensus.cc:686] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:52.918902 17046 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:52.962868 15178 rpcz_store.cc:269] 0806 20:35:50.234549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.238287 (+ 3738us) service_pool.cc:227] Handling call
0806 20:35:52.106208 (+1867921us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:52.892638 (+786430us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52852992}
I20260806 20:35:53.021288 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:53.023833 17314 raft_consensus.cc:2781] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:35:52.971249 17303 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:52.958492 17110 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:52.996749 16820 raft_consensus.cc:2781] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:53.041132 17191 consensus_queue.cc:237] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:53.043686 17046 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:53.045811 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:53.046754 15179 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 408) took 2803 ms (client timeout 1614 ms). Trace:
I20260806 20:35:53.027484 17263 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:53.024495 17314 raft_consensus.cc:686] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [term 9 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:53.053028 17314 consensus_queue.cc:237] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:53.047194 15179 rpcz_store.cc:269] 0806 20:35:50.243517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.264473 (+ 20956us) service_pool.cc:227] Handling call
0806 20:35:53.045615 (+2781142us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:53.046673 (+ 1058us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85935872}
I20260806 20:35:53.055011 17263 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.056207 16820 raft_consensus.cc:686] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:53.057225 16820 consensus_queue.cc:237] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:53.058130 14555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1703) took 2816 ms (client timeout 1767 ms). Trace:
W20260806 20:35:53.058524 14555 rpcz_store.cc:269] 0806 20:35:50.242166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.279104 (+ 36938us) service_pool.cc:227] Handling call
0806 20:35:53.057909 (+2778805us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400273dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:53.058038 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3975267072}
I20260806 20:35:53.060020 14555 raft_consensus.cc:1226] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:53.056133 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1699) took 2871 ms. Trace:
I20260806 20:35:53.065569 14550 rpcz_store.cc:276] 0806 20:35:50.184510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.557033 (+372523us) service_pool.cc:227] Handling call
0806 20:35:50.689268 (+132235us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:52.257835 (+1568567us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:53.055876 (+798041us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54000b3bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:53.056074 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3588806016}
I20260806 20:35:53.027446 17010 consensus_queue.cc:237] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:53.098160 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 460515us
I20260806 20:35:53.045822 17110 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:52.998077 17295 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.110735 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:53.111482 14547 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:53.112125 17059 consensus_queue.cc:1046] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:35:53.107051 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:53.115347 15179 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:53.116017 17219 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:53.104425 17299 log.cc:924] Time spent T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.270s user 0.000s sys 0.001s
I20260806 20:35:53.118801 14442 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:53.123723 14595 connection.cc:232] RPC call timeout handler was delayed by -1.5222761949989945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
I20260806 20:35:53.124245 14442 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:53.128867 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:53.124208 14595 connection.cc:232] RPC call timeout handler was delayed by -1.4772761950152926s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1642s for a response
W20260806 20:35:53.129734 14595 connection.cc:232] RPC call timeout handler was delayed by -1.4718761950207409s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20260806 20:35:53.130636 14595 connection.cc:232] RPC call timeout handler was delayed by -1.4358761949988548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260806 20:35:53.130908 14595 connection.cc:232] RPC call timeout handler was delayed by -1.3674761950096581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
W20260806 20:35:53.131165 14595 connection.cc:232] RPC call timeout handler was delayed by -1.3548761950223707s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
W20260806 20:35:53.131426 14595 connection.cc:232] RPC call timeout handler was delayed by -1.0524761950073298s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2114s for a response
I20260806 20:35:53.131652 17295 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:53.036674 17303 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:53.133275 17219 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:35:53.134527 17185 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:53.138177 14442 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:53.139206 17312 raft_consensus.cc:2781] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:53.139767 17312 raft_consensus.cc:686] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:53.125073 17305 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260806 20:35:53.113936 14547 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:35:53.119588 14478 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:53.149694 14626 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 20:35:53.150344 14442 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:53.154672 14626 raft_consensus.cc:1226] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:53.165457 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:53.140235 17239 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:53.167397 14628 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:53.127346 17190 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:53.172662 17190 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.172832 17059 consensus_queue.cc:1035] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:35:53.173336 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:53.150890 17113 consensus_queue.cc:1046] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260806 20:35:53.174659 17190 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:53.175280 17190 raft_consensus.cc:3032] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:53.175611 17190 raft_consensus.cc:729] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:53.170732 17263 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:53.171452 14475 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:53.165068 17185 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:53.116046 17260 consensus_queue.cc:237] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.177065 17185 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:53.181564 15179 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:53.176551 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:53.193516 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:53.194000 14628 raft_consensus.cc:2379] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 4: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:53.176911 17182 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:53.196074 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:53.196784 14633 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:53.179080 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:53.140842 17312 consensus_queue.cc:237] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:53.169605 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:53.192724 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:53.197942 14626 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:53.193213 17185 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:53.198271 14631 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:53.176424 17190 consensus_queue.cc:260] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.149482 17110 raft_consensus.cc:3032] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:35:53.200330 17110 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:53.200667 17182 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:53.201709 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:53.199754 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:53.203230 14626 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:53.204754 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:53.207019 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:53.201582 17110 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.208123 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:53.165838 17219 consensus_queue.cc:1046] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260806 20:35:53.208441 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:53.634266 14555 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.013s sys 0.062s
W20260806 20:35:53.761595 14555 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.652s user 0.018s sys 0.066s
I20260806 20:35:53.761538 14633 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:53.295060 14626 raft_consensus.cc:2454] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:53.236611 14547 raft_consensus.cc:2379] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 3: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:53.488246 14627 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:53.784915 17220 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:53.899782 16286 consensus_queue.cc:1046] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 20:35:53.250262 17181 consensus_queue.cc:1046] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.636s
I20260806 20:35:53.886885 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:53.902416 14628 raft_consensus.cc:2396] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:35:53.253643 17303 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:53.887920 14631 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:35:53.871810 17185 consensus_meta.cc:287] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.673s user 0.009s sys 0.002s
I20260806 20:35:53.209570 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:53.440553 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 331554us
I20260806 20:35:53.753350 17010 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:53.778491 17177 raft_consensus.cc:3032] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:53.249533 17313 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.636s
I20260806 20:35:53.902732 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 373) took 3798 ms. Trace:
I20260806 20:35:53.911353 14629 rpcz_store.cc:276] 0806 20:35:50.103864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.236797 (+132933us) service_pool.cc:227] Handling call
0806 20:35:50.237137 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:51.328464 (+1091327us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:53.902500 (+2574036us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540009f454. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:53.902675 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2508363136}
I20260806 20:35:53.439953 17266 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:53.900818 17219 raft_consensus.cc:3032] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:53.905802 14625 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:53.893188 17190 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:53.885717 16853 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:53.209969 17137 consensus_queue.cc:1046] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.698s
W20260806 20:35:53.917325 14478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.010s sys 0.004s
W20260806 20:35:53.925668 14478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s user 0.010s sys 0.004s
I20260806 20:35:53.902743 17155 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:53.936075 14633 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:53.957698 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1577) took 1576 ms. Trace:
I20260806 20:35:53.958128 14548 rpcz_store.cc:276] 0806 20:35:52.381376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.381526 (+ 150us) service_pool.cc:227] Handling call
0806 20:35:53.957527 (+1576001us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54003e8ad4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:53.957638 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":422565120}
I20260806 20:35:53.929039 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:53.959611 15187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 486) took 1487 ms (client timeout 1642 ms). Trace:
W20260806 20:35:53.960039 15187 rpcz_store.cc:269] 0806 20:35:52.471854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.489187 (+ 17333us) service_pool.cc:227] Handling call
0806 20:35:53.928816 (+1439629us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:53.959541 (+ 30725us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4391296}
I20260806 20:35:53.908003 17177 raft_consensus.cc:729] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:53.919884 17260 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.001s sys 0.017s
I20260806 20:35:53.916541 17219 raft_consensus.cc:729] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:53.896440 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:53.911062 17010 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:53.949268 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 187526us
W20260806 20:35:53.989640 14550 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.025s sys 0.042s
W20260806 20:35:53.990291 14550 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.855s user 0.026s sys 0.042s
I20260806 20:35:53.903637 17220 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.912935 17185 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:53.956424 14624 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
W20260806 20:35:53.963238 14632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1850) took 4509 ms (client timeout 2057 ms). Trace:
I20260806 20:35:53.963488 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:53.926339 16853 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:53.975396 13287 debug-util.cc:398] Leaking SignalData structure 0x7b0800392f00 after lost signal to thread 14518
I20260806 20:35:53.986516 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:53.987972 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1605) took 1518 ms. Trace:
I20260806 20:35:53.941205 17155 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:53.987743 17177 consensus_queue.cc:260] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:53.982676 17260 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s user 0.002s sys 0.018s
I20260806 20:35:53.994707 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:53.995752 14632 rpcz_store.cc:269] 0806 20:35:49.446585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.452016 (+ 5431us) service_pool.cc:227] Handling call
0806 20:35:53.955655 (+4503639us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b540020e254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:53.955882 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1628907776}
I20260806 20:35:53.997576 14476 rpcz_store.cc:276] 0806 20:35:52.469071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.981652 (+512581us) service_pool.cc:227] Handling call
0806 20:35:52.981979 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:53.001307 (+ 19328us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:53.001350 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:53.116041 (+114691us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b2c0006bf28. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:53.116371 (+ 330us) log.cc:842] Serialized 26 byte log entry
0806 20:35:53.594155 (+477784us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:53.616045 (+ 21890us) log.cc:842] Serialized 14 byte log entry
0806 20:35:53.780633 (+164588us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:53.780720 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:53.987597 (+206877us) raft_consensus.cc:1671] finished
0806 20:35:53.987731 (+ 134us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:53.987899 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":183054,"prepare.queue_time_us":186,"prepare.run_wall_time_us":290862,"raft.queue_time_us":185548,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":7659,"spinlock_wait_cycles":271389952,"thread_start_us":477317,"threads_started":1,"wal-append.queue_time_us":868465}
W20260806 20:35:53.997404 13287 kernel_stack_watchdog.cc:198] Thread 14518 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/proxy.cc:249 for 304ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260806 20:35:54.004745 14548 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:54.006920 16853 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.007212 14547 pending_rounds.cc:76] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 2
I20260806 20:35:54.022899 14547 pending_rounds.cc:97] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 20:35:54.023258 14547 raft_consensus.cc:2866] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:54.020378 14476 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 20:35:54.022183 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1591) took 1564 ms. Trace:
I20260806 20:35:54.024775 14548 rpcz_store.cc:276] 0806 20:35:52.458136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.000543 (+1542407us) service_pool.cc:227] Handling call
0806 20:35:54.000917 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.021809 (+ 20892us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.022109 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:54.036420 17177 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:54.058491 17266 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:54.033879 17219 consensus_queue.cc:260] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:54.075299 14551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1683) took 3895 ms (client timeout 1688 ms). Trace:
W20260806 20:35:54.075904 14551 rpcz_store.cc:269] 0806 20:35:50.179593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.233148 (+ 53555us) service_pool.cc:227] Handling call
0806 20:35:54.075001 (+3841853us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54001faed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:54.075202 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3578746240}
I20260806 20:35:54.088862 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1601) took 1126 ms. Trace:
I20260806 20:35:54.089802 14551 rpcz_store.cc:276] 0806 20:35:52.962665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.079138 (+1116473us) service_pool.cc:227] Handling call
0806 20:35:54.088161 (+ 9023us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.088420 (+ 259us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.088476 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.088536 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:54.088583 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.088651 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.088784 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:35:54.094141 14479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.009s sys 0.072s
W20260806 20:35:54.095762 14479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s user 0.009s sys 0.072s
W20260806 20:35:54.093865 14477 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.022s sys 0.030s
W20260806 20:35:54.105197 14477 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s user 0.022s sys 0.030s
I20260806 20:35:54.091455 17219 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:54.093935 17185 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.110957 17010 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:54.136854 17155 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:54.141940 17220 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.180630 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 397) took 1016 ms. Trace:
I20260806 20:35:54.181066 14627 rpcz_store.cc:276] 0806 20:35:53.164722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.170323 (+ 5601us) service_pool.cc:227] Handling call
0806 20:35:53.170591 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:53.901202 (+730611us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.180358 (+279156us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54003bea54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:54.180571 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729934080}
I20260806 20:35:54.183974 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 369) took 5038 ms. Trace:
W20260806 20:35:54.163664 17321 log.cc:924] Time spent T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
I20260806 20:35:54.184360 14624 rpcz_store.cc:276] 0806 20:35:49.145156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.959000 (+813844us) service_pool.cc:227] Handling call
0806 20:35:53.956178 (+3997178us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:53.995386 (+ 39208us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.183744 (+188358us) spinlock_profiling.cc:243] Waited 4 s on lock 0x7b540020e250. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:54.183886 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9192067840}
I20260806 20:35:54.189097 17160 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.190016 17160 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.200920 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:54.169472 17273 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:54.211364 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 308033us
I20260806 20:35:54.160523 17010 consensus_queue.cc:1046] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260806 20:35:54.228549 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:54.232268 14633 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:54.232400 14624 raft_consensus.cc:1261] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:54.227283 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:54.267107 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 45316.1us
I20260806 20:35:54.267851 14632 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:54.269089 17302 consensus_queue.cc:1046] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 20:35:54.269879 15151 leader_election.cc:334] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.774s (SENT)
I20260806 20:35:54.270324 15151 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:54.271433 15151 leader_election.cc:334] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.819s (SENT)
W20260806 20:35:54.272245 15151 leader_election.cc:334] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.977s (SENT)
I20260806 20:35:54.272595 15151 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:54.272786 17302 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:54.273460 17302 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:35:54.273563 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:54.274528 17010 consensus_queue.cc:1225] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer 148bf2a11dba423ab364371f0458e871 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260806 20:35:54.276053 17326 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260806 20:35:54.276983 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:54.277498 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 359) took 1274 ms. Trace:
I20260806 20:35:54.277886 14478 rpcz_store.cc:276] 0806 20:35:53.003343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.148380 (+145037us) service_pool.cc:227] Handling call
0806 20:35:53.149371 (+ 991us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:53.149833 (+ 462us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:53.149859 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:53.182165 (+ 32306us) log.cc:842] Serialized 52 byte log entry
0806 20:35:53.932188 (+750023us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:53.932241 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:53.932331 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:54.277293 (+344962us) raft_consensus.cc:1671] finished
0806 20:35:54.277311 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.277442 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":90,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":11904,"thread_start_us":743631,"threads_started":1,"wal-append.queue_time_us":961596}
I20260806 20:35:54.284879 17338 consensus_queue.cc:1046] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:35:54.286561 17302 raft_consensus.cc:3032] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:54.286950 17302 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:54.267194 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 53117.8us
I20260806 20:35:54.294039 14444 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:54.294237 14633 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:54.295063 14441 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:54.299511 14441 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.300894 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:54.294435 17321 log.cc:924] Time spent T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260806 20:35:54.303584 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:54.230595 17155 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.305773 14444 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:54.318804 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 324167us
I20260806 20:35:54.301623 17160 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.321286 17248 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:54.302484 16853 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:54.310187 17227 raft_consensus.cc:3037] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:54.310738 17339 consensus_queue.cc:1046] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:35:54.312026 17333 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:54.315363 17335 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:54.239101 17337 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:54.289201 17260 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:54.317957 14443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.018s sys 0.007s
W20260806 20:35:54.409909 14443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.018s sys 0.007s
I20260806 20:35:54.287806 17302 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.319301 14444 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.307379 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:54.321316 15181 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:54.304204 17331 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.375948 17248 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:54.319550 17266 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:35:54.215899 17273 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.375705 14595 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.362751 17137 consensus_queue.cc:1225] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer 148bf2a11dba423ab364371f0458e871 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260806 20:35:54.409392 17260 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.320040 17155 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:54.308156 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.002s sys 0.022s
W20260806 20:35:54.423930 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.662s user 0.002s sys 0.022s
I20260806 20:35:54.424775 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1602) took 1960 ms. Trace:
I20260806 20:35:54.414343 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1606) took 1943 ms. Trace:
I20260806 20:35:54.386308 17335 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:54.412412 17339 consensus_queue.cc:1225] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer df1245ecd2034e11ae6cf688c1c10a4d log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260806 20:35:54.447539 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 148210us
W20260806 20:35:54.450037 15151 leader_election.cc:334] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.089s (SENT)
W20260806 20:35:54.451385 15151 leader_election.cc:334] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.163s (SENT)
I20260806 20:35:54.451881 15151 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:54.425199 14479 rpcz_store.cc:276] 0806 20:35:52.464634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.980112 (+515478us) service_pool.cc:227] Handling call
0806 20:35:52.980410 (+ 298us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:53.001194 (+ 20784us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:53.001237 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:53.113714 (+112477us) log.cc:842] Serialized 26 byte log entry
0806 20:35:54.097755 (+984041us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:54.421554 (+323799us) log.cc:842] Serialized 14 byte log entry
0806 20:35:54.424380 (+ 2826us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.424514 (+ 134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:54.424560 (+ 46us) raft_consensus.cc:1671] finished
0806 20:35:54.424574 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.424714 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22872,"raft.queue_time_us":21197,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":1243417,"threads_started":2,"wal-append.queue_time_us":1075615}
I20260806 20:35:54.454020 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:54.454310 14627 raft_consensus.cc:1261] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:35:54.455526 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:54.456198 17335 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:54.456980 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:54.457931 14479 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:54.460321 14479 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:54.417531 17248 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:54.428483 14478 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 20:35:54.419399 17332 raft_consensus.cc:3032] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:54.421221 17302 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:54.423759 17137 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:54.483229 17314 raft_consensus.cc:2726] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:35:54.484786 14469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1590) took 4060 ms (client timeout 1592 ms). Trace:
W20260806 20:35:54.485232 17336 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
W20260806 20:35:54.487478 17327 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260806 20:35:54.488374 14549 raft_consensus.cc:1226] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260806 20:35:54.489089 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1713) took 2941 ms. Trace:
I20260806 20:35:54.489588 17290 raft_consensus.cc:3032] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:54.490085 17290 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:54.490995 17290 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.491986 17290 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:54.489656 17330 log.cc:924] Time spent T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
W20260806 20:35:54.495560 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 44219.1us
I20260806 20:35:54.496840 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:53.127012 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:35:54.498687 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1702) took 4310 ms. Trace:
I20260806 20:35:54.479074 17332 raft_consensus.cc:729] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:54.481849 15178 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
W20260806 20:35:54.485251 14469 rpcz_store.cc:269] 0806 20:35:50.424656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.462133 (+ 37477us) service_pool.cc:227] Handling call
0806 20:35:54.484592 (+4022459us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54003327d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:54.484709 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4726471040}
I20260806 20:35:54.507820 14478 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260806 20:35:54.489449 14549 rpcz_store.cc:276] 0806 20:35:51.547870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.470887 (+2923017us) service_pool.cc:227] Handling call
0806 20:35:54.488148 (+ 17261us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:54.488833 (+ 685us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.489033 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:54.510883 14479 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:35:54.516486 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:54.517663 17241 raft_consensus.cc:686] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:54.489096 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 360) took 1335 ms. Trace:
I20260806 20:35:54.520025 14477 rpcz_store.cc:276] 0806 20:35:53.153277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.183079 (+ 29802us) service_pool.cc:227] Handling call
0806 20:35:53.183461 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:53.183795 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:53.183834 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:53.184366 (+ 532us) log.cc:842] Serialized 26 byte log entry
0806 20:35:54.105869 (+921503us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:54.105925 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.106037 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:54.488810 (+382773us) raft_consensus.cc:1671] finished
0806 20:35:54.488837 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.489019 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1092,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":1353,"thread_start_us":921057,"threads_started":1,"wal-append.queue_time_us":1116461}
W20260806 20:35:54.520648 15151 leader_election.cc:334] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.181s (SENT)
I20260806 20:35:54.523041 15151 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:54.471302 14474 rpcz_store.cc:276] 0806 20:35:52.470919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.000201 (+529282us) service_pool.cc:227] Handling call
0806 20:35:53.110094 (+109893us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:53.110781 (+ 687us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:35:53.111390 (+ 609us) log.cc:842] Serialized 14 byte log entry
0806 20:35:53.260968 (+149578us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:53.413538 (+152570us) log.cc:842] Serialized 26 byte log entry
0806 20:35:53.413738 (+ 200us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:53.413787 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:53.413889 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:54.414059 (+1000170us) raft_consensus.cc:1671] finished
0806 20:35:54.414089 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.414266 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":292958,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":2058,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":143980,"thread_start_us":149124,"threads_started":1,"wal-append.queue_time_us":881708}
I20260806 20:35:54.524762 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1724) took 2464 ms. Trace:
W20260806 20:35:54.530159 15151 leader_election.cc:334] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.185s (SENT)
I20260806 20:35:54.530752 15151 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:54.499214 14552 rpcz_store.cc:276] 0806 20:35:50.188830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:51.249880 (+1061050us) service_pool.cc:227] Handling call
0806 20:35:51.250190 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:51.250391 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:51.425545 (+175154us) log.cc:842] Serialized 14 byte log entry
0806 20:35:51.517374 (+ 91829us) log.cc:842] Serialized 14 byte log entry
0806 20:35:52.310064 (+792690us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b18002231e0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b0598f96 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11
0806 20:35:52.476716 (+166652us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b18002231e0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:35:54.498110 (+2021394us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b18002231e0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05424ca 00007fb9b9c23f99 00007fb9b0406990 00007fb9b05c8864 00007fb9b05c8155 00007fb9b34eb3d0 00007fb9b34fd576 00007fb9b350e0d7 00007fb9b350e058
0806 20:35:54.498268 (+ 158us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.498336 (+ 68us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:54.498383 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.498452 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.498612 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2966515,"prepare.queue_time_us":198562,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":108,"raft.queue_time_us":2994999,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":4571,"spinlock_wait_cycles":20803072,"thread_start_us":217368,"threads_started":2,"wal-append.queue_time_us":93236,"wal-append.run_cpu_time_us":3548,"wal-append.run_wall_time_us":1505069}
W20260806 20:35:54.530864 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 211649us
I20260806 20:35:54.532388 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.534507 17103 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:35:54.533250 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 3.007e+06us
I20260806 20:35:54.536835 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:54.538892 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1600) took 1620 ms. Trace:
I20260806 20:35:54.540630 14518 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:54.528065 14554 rpcz_store.cc:276] 0806 20:35:52.060214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.235205 (+174991us) service_pool.cc:227] Handling call
0806 20:35:52.257186 (+ 21981us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:52.257365 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:52.360090 (+102725us) log.cc:842] Serialized 14 byte log entry
0806 20:35:54.524352 (+2164262us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.524441 (+ 89us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:54.524516 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.524584 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.524709 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2164050,"prepare.queue_time_us":119131,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":2164217,"thread_start_us":71867,"threads_started":1}
I20260806 20:35:54.539274 14548 rpcz_store.cc:276] 0806 20:35:52.918822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.025811 (+1106989us) service_pool.cc:227] Handling call
0806 20:35:54.026181 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.026432 (+ 251us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:54.041605 (+ 15173us) log.cc:842] Serialized 14 byte log entry
0806 20:35:54.538563 (+496958us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.538616 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:54.538668 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.538729 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.538837 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":307533,"prepare.queue_time_us":8399,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":188929,"threads_started":1,"wal-append.queue_time_us":341674}
I20260806 20:35:54.545414 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:54.547045 17194 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:54.547333 14444 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:54.547478 17194 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:54.548022 17194 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:54.550137 16853 raft_consensus.cc:686] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:54.550616 16942 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:35:54.550832 17349 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:54.551239 16853 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:54.552093 15151 leader_election.cc:334] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.185s (SENT)
I20260806 20:35:54.556807 14444 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
W20260806 20:35:54.554369 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 137064us
I20260806 20:35:54.549824 17335 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.559971 16942 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:54.560316 14478 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 20:35:54.560892 17335 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.560402 16942 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:54.560259 14444 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.561834 16942 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:54.562381 17180 consensus_queue.cc:1046] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:35:54.562788 17266 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:54.563265 17266 raft_consensus.cc:686] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:54.563426 17256 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:54.563907 17256 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.564204 17266 consensus_queue.cc:237] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.565348 14444 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.566094 17256 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.567090 17256 consensus_queue.cc:1035] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 20:35:54.525102 17324 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260806 20:35:54.556169 15151 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:54.570448 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:54.571451 14552 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 8: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:35:54.573397 15151 leader_election.cc:334] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.249s (SENT)
I20260806 20:35:54.644992 14441 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:54.417251 17331 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.652959 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:54.653126 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:54.533272 17234 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:54.653743 14549 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:54.544387 17175 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.662482 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:54.667351 14552 raft_consensus.cc:2379] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:54.524582 17343 raft_consensus.cc:2781] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:54.670799 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:54.671770 14554 raft_consensus.cc:2379] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:54.676206 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1733) took 1554 ms. Trace:
I20260806 20:35:54.677219 14554 rpcz_store.cc:276] 0806 20:35:53.121644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.672421 (+1550777us) service_pool.cc:227] Handling call
0806 20:35:54.674788 (+ 2367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.675439 (+ 651us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.675504 (+ 65us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.675628 (+ 124us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:54.675710 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.675841 (+ 131us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.676063 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:54.646085 16844 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:35:54.648834 17335 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:54.505013 17344 consensus_queue.cc:1046] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.186s
I20260806 20:35:54.529496 17181 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.135s
I20260806 20:35:54.503710 17337 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.159s
W20260806 20:35:54.664672 17190 consensus_meta.cc:287] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.742s user 0.010s sys 0.003s
I20260806 20:35:54.683696 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1734) took 1537 ms. Trace:
W20260806 20:35:54.691510 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 125116us
I20260806 20:35:54.719676 14554 rpcz_store.cc:276] 0806 20:35:53.146477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.682436 (+1535959us) service_pool.cc:227] Handling call
0806 20:35:54.683096 (+ 660us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.683289 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.683326 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.683376 (+ 50us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:54.683414 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.683482 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.683604 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:54.723057 14444 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:54.697410 14552 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:54.725178 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1732) took 1603 ms. Trace:
I20260806 20:35:54.725499 14552 rpcz_store.cc:276] 0806 20:35:53.121327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.668020 (+1546693us) service_pool.cc:227] Handling call
0806 20:35:54.697177 (+ 29157us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.724834 (+ 27657us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.725124 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:54.665510 17137 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.700870 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1731) took 1579 ms. Trace:
I20260806 20:35:54.726883 14549 rpcz_store.cc:276] 0806 20:35:53.120965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.663401 (+1542436us) service_pool.cc:227] Handling call
0806 20:35:54.700281 (+ 36880us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.700509 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.700541 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.700587 (+ 46us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:54.700627 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.700696 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.700814 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:54.680812 17175 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.730662 17175 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.704859 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1695) took 4521 ms. Trace:
I20260806 20:35:54.731289 14546 rpcz_store.cc:276] 0806 20:35:50.183864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:50.315535 (+131671us) service_pool.cc:227] Handling call
0806 20:35:50.352267 (+ 36732us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:51.290092 (+937825us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:35:52.060250 (+770158us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b18003bb480. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b318282c 00007fb9b05ad9dc 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05c8155 00007fb9b6ec8167 00007fb9b34fd807 00007fb9b350cf0f
0806 20:35:52.060378 (+ 128us) log.cc:842] Serialized 14 byte log entry
0806 20:35:54.517929 (+2457551us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:54.518427 (+ 498us) log.cc:842] Serialized 26 byte log entry
0806 20:35:54.553648 (+ 35221us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:54.553698 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.553777 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:54.704519 (+150742us) raft_consensus.cc:1671] finished
0806 20:35:54.704636 (+ 117us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.704796 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2368706,"prepare.queue_time_us":167495,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":38,"raft.queue_time_us":3317656,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":52861,"spinlock_wait_cycles":3917056,"thread_start_us":1151498,"threads_started":4,"wal-append.queue_time_us":323160,"wal-append.run_cpu_time_us":1448,"wal-append.run_wall_time_us":1002180}
I20260806 20:35:54.662726 17166 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.735227 17166 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.737286 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1736) took 1527 ms. Trace:
I20260806 20:35:54.737963 14549 rpcz_store.cc:276] 0806 20:35:53.210285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.727853 (+1517568us) service_pool.cc:227] Handling call
0806 20:35:54.728269 (+ 416us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.736843 (+ 8574us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.736889 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.736948 (+ 59us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:54.736995 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.737074 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:54.737220 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:54.747961 14549 raft_consensus.cc:3032] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:54.747143 17166 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:54.669413 17353 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:54.752741 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:54.750699 14549 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:54.759938 14624 raft_consensus.cc:3032] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:54.760404 14624 raft_consensus.cc:729] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:54.753418 14554 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:54.763550 14554 raft_consensus.cc:1261] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:54.764256 14629 raft_consensus.cc:1261] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:35:54.764901 14554 raft_consensus.cc:1261] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:35:54.765115 14632 raft_consensus.cc:1261] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:54.766791 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:54.767576 14628 raft_consensus.cc:3037] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:54.714504 17335 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:35:54.704048 16942 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.762868 17353 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:54.518692 17241 consensus_queue.cc:237] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.774647 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:54.780629 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 251308us
I20260806 20:35:54.752462 15181 raft_consensus.cc:1261] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:35:54.761389 14624 consensus_queue.cc:260] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.717729 17343 raft_consensus.cc:686] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:54.775622 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:54.781111 14629 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:35:54.767973 17341 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.799366 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:54.754724 14549 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.808101 14549 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:54.779402 17335 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:54.816104 14628 raft_consensus.cc:2454] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:54.772795 17235 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.715072 17194 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.798256 17111 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:54.826711 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:54.782290 17353 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:54.783988 17260 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:54.797854 17354 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:54.829725 17347 log.cc:924] Time spent T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
W20260806 20:35:54.830642 17350 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260806 20:35:54.810720 17256 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.809365 17234 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.781248 17355 consensus_queue.cc:1046] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260806 20:35:54.680294 17332 consensus_queue.cc:260] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.838374 17194 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.834676 14473 raft_consensus.cc:1261] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260806 20:35:54.828393 16942 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.848050 14633 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:54.903462 14473 pending_rounds.cc:76] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 6
I20260806 20:35:54.904665 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1511) took 2726 ms. Trace:
I20260806 20:35:54.818243 17155 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:54.854065 14595 leader_election.cc:398] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:54.908915 14595 leader_election.cc:401] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260806 20:35:54.909698 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 55801.8us
I20260806 20:35:54.805302 17344 raft_consensus.cc:3032] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Stepping down as leader of term 4
W20260806 20:35:54.847879 17326 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260806 20:35:54.914661 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:54.915338 14633 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:54.818763 17273 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:54.861662 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:35:54.870617 17219 consensus_meta.cc:287] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.728s user 0.011s sys 0.002s
I20260806 20:35:54.863780 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 349) took 2433 ms. Trace:
I20260806 20:35:54.924947 14470 rpcz_store.cc:276] 0806 20:35:52.430559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.430685 (+ 126us) service_pool.cc:227] Handling call
0806 20:35:52.443815 (+ 13130us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:52.973523 (+529708us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.863517 (+1889994us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54001180d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:54.863715 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1216291584}
I20260806 20:35:54.848027 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:54.763954 17137 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.905315 14473 pending_rounds.cc:97] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260806 20:35:54.929144 14473 raft_consensus.cc:2866] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:54.851856 17302 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.906216 14630 rpcz_store.cc:276] 0806 20:35:52.178348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.250919 (+ 72571us) service_pool.cc:227] Handling call
0806 20:35:52.251217 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:52.277638 (+ 26421us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.904383 (+2626745us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b5400312754. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:54.904598 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11618560}
I20260806 20:35:54.933441 17331 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.934042 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:54.934715 14630 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:54.934983 17359 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260806 20:35:54.862918 17234 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.911041 14595 leader_election.cc:398] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:54.936781 14595 leader_election.cc:401] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 20:35:54.937000 17355 raft_consensus.cc:3032] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:54.937412 17355 raft_consensus.cc:729] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:54.937618 15185 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:54.938033 14595 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:54.938261 17355 consensus_queue.cc:260] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.939348 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.906454 17248 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:54.945658 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 82070.7us
I20260806 20:35:54.818377 17341 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.917480 14628 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
W20260806 20:35:54.858526 17357 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260806 20:35:54.943192 14549 raft_consensus.cc:1261] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260806 20:35:54.851393 17111 raft_consensus.cc:686] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:54.939640 14633 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:54.959385 14549 pending_rounds.cc:76] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 3
I20260806 20:35:54.951659 14630 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:54.912398 17344 raft_consensus.cc:729] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:54.959534 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:54.856567 17354 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.960538 17220 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.861348 17256 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:54.916708 17332 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:54.954453 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 249732us
I20260806 20:35:54.974944 14444 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:54.961530 14549 pending_rounds.cc:97] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:35:54.981784 14549 raft_consensus.cc:2866] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:54.929004 17273 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:54.939586 17355 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:54.993412 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:54.963093 17219 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:54.961521 17377 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:55.004972 17219 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:54.963976 17111 consensus_queue.cc:237] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.973626 15182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 478) took 2513 ms. Trace:
I20260806 20:35:54.970081 17137 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:54.965201 17354 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:54.972180 17256 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:54.985114 17266 raft_consensus.cc:2781] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:55.021831 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:55.021863 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 82591.9us
I20260806 20:35:54.945986 17352 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
W20260806 20:35:55.021601 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 307648us
I20260806 20:35:54.966471 17220 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.028020 14595 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:35:55.026501 14444 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:54.958320 17302 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.016265 17219 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:55.029981 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:54.987923 17335 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.031189 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:55.032974 14444 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:54.964705 17363 consensus_queue.cc:1035] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260806 20:35:55.033020 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:55.045315 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:55.045887 14554 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:55.031121 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:55.032258 17376 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5.
I20260806 20:35:55.035138 17256 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:55.035429 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:55.020782 15182 rpcz_store.cc:276] 0806 20:35:52.460092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.467816 (+ 7724us) service_pool.cc:227] Handling call
0806 20:35:52.652699 (+184883us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:52.883096 (+230397us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:54.973263 (+2090167us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b54004d08d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:54.973513 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":212072832}
I20260806 20:35:55.030548 17235 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:55.047662 14630 raft_consensus.cc:2396] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:35:54.916594 17343 consensus_queue.cc:237] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:54.992911 17273 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:55.056180 17360 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.071s user 0.002s sys 0.000s
I20260806 20:35:55.027714 17137 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:55.057283 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:55.066327 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:55.066771 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:55.085168 17375 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:55.086308 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 56409.4us
I20260806 20:35:55.033927 17266 raft_consensus.cc:686] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:54.962443 17344 consensus_queue.cc:260] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.060312 17235 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:55.222445 14631 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:55.222904 17235 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:55.223331 17137 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:55.058132 17354 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:55.272178 14549 raft_consensus.cc:2396] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260806 20:35:55.272924 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 285619us
I20260806 20:35:55.226562 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 356) took 2793 ms. Trace:
I20260806 20:35:55.283933 14473 rpcz_store.cc:276] 0806 20:35:52.433412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.455324 (+ 21912us) service_pool.cc:227] Handling call
0806 20:35:52.469390 (+ 14066us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.960322 (+2490932us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.211644 (+251322us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b54003c4454. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:55.226476 (+ 14832us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5438884608}
I20260806 20:35:55.224041 17137 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:55.298069 14595 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:55.300354 14633 raft_consensus.cc:3032] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:55.308557 14633 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:55.068779 17256 raft_consensus.cc:686] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:55.163245 17285 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:55.309392 14633 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.325801 14633 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:55.312201 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:55.326417 14628 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:55.311738 17352 consensus_queue.cc:1035] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:55.313714 17344 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:55.314687 17384 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:55.285437 17354 raft_consensus.cc:686] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:55.316709 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:55.176062 17374 consensus_queue.cc:1046] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.351s
I20260806 20:35:55.337919 14442 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:55.277827 17375 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:55.271445 17302 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:55.352994 17354 consensus_queue.cc:237] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.338508 17374 raft_consensus.cc:3032] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:55.312652 16853 consensus_queue.cc:1035] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:35:55.277108 17248 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:55.344914 17375 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:55.345619 17147 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:55.346495 17384 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:55.355564 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:55.362006 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:35:55.343190 17290 consensus_meta.cc:287] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.851s user 0.015s sys 0.000s
I20260806 20:35:55.370927 14554 raft_consensus.cc:3032] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:55.371493 14554 raft_consensus.cc:729] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:55.343171 17312 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:55.354730 17374 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:55.357777 14628 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:55.358155 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:55.338483 17386 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:55.372989 14629 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:55.329126 17363 consensus_queue.cc:1035] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260806 20:35:55.374368 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1586) took 2944 ms. Trace:
W20260806 20:35:55.357048 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 3.60169e+06us
I20260806 20:35:55.390561 14521 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:55.284353 17376 consensus_queue.cc:1159] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" responder_term: 4 status { last_received { term: 4 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 4 index: 6 } }
W20260806 20:35:55.380776 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 840194us
I20260806 20:35:55.368340 17384 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:55.386890 14444 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:55.391426 14554 consensus_queue.cc:260] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.394702 14554 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:55.401211 17312 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:55.401902 17312 raft_consensus.cc:686] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:55.403019 17312 consensus_queue.cc:237] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.403852 17384 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.409263 14633 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260806 20:35:55.409842 14633 pending_rounds.cc:76] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 6
I20260806 20:35:55.388835 14555 rpcz_store.cc:276] 0806 20:35:52.429845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.061785 (+631940us) service_pool.cc:227] Handling call
0806 20:35:53.062099 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:53.062338 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:35:53.762224 (+699886us) log.cc:842] Serialized 14 byte log entry
0806 20:35:55.374045 (+1611821us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:55.374105 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:55.374157 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.374215 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:55.374318 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1219969,"prepare.queue_time_us":881309,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"thread_start_us":1044106,"threads_started":2,"wal-append.queue_time_us":542314}
I20260806 20:35:55.422111 14633 pending_rounds.cc:97] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 20:35:55.422663 14633 raft_consensus.cc:2866] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:55.424254 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:55.425148 14633 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:55.360386 17266 consensus_queue.cc:237] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.373685 17147 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:55.348573 17285 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.379884 17197 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:55.441360 14554 raft_consensus.cc:1261] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:35:55.447327 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1529) took 1221 ms. Trace:
I20260806 20:35:55.384963 17103 raft_consensus.cc:686] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:55.447834 14627 rpcz_store.cc:276] 0806 20:35:54.225998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.226121 (+ 123us) service_pool.cc:227] Handling call
0806 20:35:54.226406 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.454846 (+228440us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.447041 (+992195us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400180454. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:55.447255 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":522941184}
I20260806 20:35:55.454701 14627 raft_consensus.cc:1261] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:55.460819 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1608) took 1571 ms. Trace:
I20260806 20:35:55.461339 14546 rpcz_store.cc:276] 0806 20:35:53.889224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.733853 (+844629us) service_pool.cc:227] Handling call
0806 20:35:54.734263 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:55.447650 (+713387us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:55.447696 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:55.460413 (+ 12717us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:55.460501 (+ 88us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.460588 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:55.460744 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":695185,"raft.queue_time_us":707897}
I20260806 20:35:55.411562 17335 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:55.433446 17175 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:55.439814 17147 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:55.524662 17273 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:55.578066 17387 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
W20260806 20:35:55.578883 17356 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.781s user 0.000s sys 0.009s
I20260806 20:35:55.553625 17332 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:35:55.453809 17185 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.117s
I20260806 20:35:55.412664 17256 consensus_queue.cc:237] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.556515 17175 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:55.608194 17303 raft_consensus.cc:3032] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:55.608916 17303 raft_consensus.cc:729] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:55.610369 17303 consensus_queue.cc:260] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.563382 17191 consensus_queue.cc:1035] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 20:35:55.565305 17335 consensus_queue.cc:1035] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
W20260806 20:35:55.599972 17353 consensus_meta.cc:287] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.748s user 0.009s sys 0.006s
I20260806 20:35:55.634131 14633 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
W20260806 20:35:55.635624 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 337640us
I20260806 20:35:55.641584 14595 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:55.617667 17175 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:55.649006 14595 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:55.650480 14595 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:55.651924 14595 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:55.632565 17384 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:55.653307 17392 log.cc:924] Time spent T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.190s user 0.001s sys 0.000s
I20260806 20:35:55.670476 17341 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:55.670651 17395 raft_consensus.cc:2781] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:55.673269 17160 raft_consensus.cc:3032] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:35:55.681798 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35910 (request call id 346) took 6350 ms (client timeout 2163 ms). Trace:
W20260806 20:35:55.684669 14475 rpcz_store.cc:269] 0806 20:35:49.331072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:49.440623 (+109551us) service_pool.cc:227] Handling call
0806 20:35:55.681526 (+6240903us) spinlock_profiling.cc:243] Waited 3.71 s on lock 0x7b54002ae754. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:55.681704 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8535509248}
I20260806 20:35:55.660086 16910 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:35:55.705731 17393 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:55.627764 17103 consensus_queue.cc:237] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.669101 17332 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:35:55.637939 17353 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.714686 17182 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:55.715195 17353 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:55.717504 17137 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.689455 17160 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:55.720889 17220 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:55.758059 17355 consensus_meta.cc:287] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.766s user 0.013s sys 0.000s
W20260806 20:35:55.765802 17235 consensus_meta.cc:287] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.542s user 0.008s sys 0.008s
I20260806 20:35:55.767216 17235 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.772233 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1538) took 1013 ms. Trace:
I20260806 20:35:55.772929 14629 rpcz_store.cc:276] 0806 20:35:54.758414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.799003 (+ 40589us) service_pool.cc:227] Handling call
0806 20:35:55.772009 (+973006us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54002c1354. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:55.772151 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1307495168}
I20260806 20:35:55.772087 17160 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.801162 17329 raft_consensus.cc:3032] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:35:55.801990 17329 raft_consensus.cc:729] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:55.774685 17160 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:55.803483 17329 consensus_queue.cc:260] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.816749 17234 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:55.784603 17377 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.761s user 0.014s sys 0.000s
I20260806 20:35:55.818228 17377 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:55.752115 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:56934 (request call id 3) took 1520 ms. Trace:
I20260806 20:35:55.821290 15180 rpcz_store.cc:276] 0806 20:35:54.231740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.237509 (+ 5769us) service_pool.cc:227] Handling call
0806 20:35:55.751632 (+1514123us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b54007003d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f97c5 00007fb9b34f1c1a 00007fb9b84356d9 00007fb9b1b1f184 00007fb9b1b1f112 00007fb9b1b1f041 00007fb9b1b1efbc 00007fb9b1b1e272 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:55.751976 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195442944}
I20260806 20:35:55.823411 17110 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:35:55.825357 17219 consensus_meta.cc:287] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.771s user 0.013s sys 0.000s
I20260806 20:35:55.826049 17219 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.828325 17219 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:55.839139 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56934 (request call id 2) took 1610 ms. Trace:
I20260806 20:35:55.839668 15187 rpcz_store.cc:276] 0806 20:35:54.228305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.228427 (+ 122us) service_pool.cc:227] Handling call
0806 20:35:55.838328 (+1609901us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54007003d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478af0 00007fb9b34fbf2a 00007fb9b8438c6e 00007fb9b1b2f807 00007fb9b1b2f792 00007fb9b1b2f6c1 00007fb9b1b2f63c 00007fb9b1b2e8f2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:55.839053 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269537024}
W20260806 20:35:55.841769 14553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1598) took 3138 ms (client timeout 2276 ms). Trace:
W20260806 20:35:55.842312 14553 rpcz_store.cc:269] 0806 20:35:52.703432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:52.966889 (+263457us) service_pool.cc:227] Handling call
0806 20:35:55.841475 (+2874586us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b5400356954. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:55.841644 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5508061952}
W20260806 20:35:55.850291 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 459755us
I20260806 20:35:55.850574 17241 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:35:55.850353 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 457710us
I20260806 20:35:55.854139 14521 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:55.856202 14625 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
W20260806 20:35:55.859130 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 996592us
I20260806 20:35:55.865342 17339 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:55.869138 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:55.869758 14629 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260806 20:35:55.798549 17235 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:55.887712 17378 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:55.808881 17147 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.895742 17378 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:55.902539 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 232614us
I20260806 20:35:55.903733 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1593) took 3432 ms. Trace:
I20260806 20:35:55.904134 14595 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:55.909844 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 880262us
I20260806 20:35:55.835608 17285 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:55.866155 17339 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:55.912566 17395 raft_consensus.cc:686] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:55.904213 14547 rpcz_store.cc:276] 0806 20:35:52.471044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.006199 (+1535155us) service_pool.cc:227] Handling call
0806 20:35:54.007011 (+ 812us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:54.533995 (+526984us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.534023 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:54.534558 (+ 535us) log.cc:842] Serialized 52 byte log entry
0806 20:35:54.690300 (+155742us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:54.994833 (+304533us) log.cc:842] Serialized 14 byte log entry
0806 20:35:55.442974 (+448141us) log.cc:842] Serialized 14 byte log entry
0806 20:35:55.707660 (+264686us) spinlock_profiling.cc:243] Waited 68.1 ms on lock 0x7b48001f0158. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b839a7f1 00007fb9b83918e7 00007fb9b839177a 00007fb9b848595f 00007fb9b8485627 00007fb9b8499b2c 00007fb9b8499ad8 00007fb9b8499a58 00007fb9b8499a13 00007fb9b8498cbc 00007fb9b741abe6
0806 20:35:55.849805 (+142145us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b18001362a0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05424ca 00007fb9b9c23f99 00007fb9b739bbcf 00007fb9b05cb7b6 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d
0806 20:35:55.903331 (+ 53526us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.903431 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:55.903494 (+ 63us) raft_consensus.cc:1671] finished
0806 20:35:55.903511 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:55.903666 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1601316,"prepare.queue_time_us":529202,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":26388,"raft.queue_time_us":1768137,"raft.run_cpu_time_us":2446,"raft.run_wall_time_us":251942,"spinlock_wait_cycles":163227776,"thread_start_us":426557,"threads_started":2,"wal-append.queue_time_us":437512}
W20260806 20:35:55.924365 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.003s sys 0.032s
W20260806 20:35:55.934985 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.003s sys 0.033s
W20260806 20:35:55.935313 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 660636us
I20260806 20:35:55.937776 17137 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:55.941856 14595 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:55.909924 17393 raft_consensus.cc:686] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:55.945286 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1587) took 3508 ms. Trace:
I20260806 20:35:55.938587 17180 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:55.909724 16910 raft_consensus.cc:686] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:55.910123 17332 raft_consensus.cc:686] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:55.948784 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:55.907064 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 490) took 1393 ms. Trace:
W20260806 20:35:55.949836 14471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.029s sys 0.053s
W20260806 20:35:55.950742 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 97598.6us
W20260806 20:35:55.951769 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 97677.9us
I20260806 20:35:55.945776 14550 rpcz_store.cc:276] 0806 20:35:52.436576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.066658 (+630082us) service_pool.cc:227] Handling call
0806 20:35:53.119518 (+ 52860us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:53.119763 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:35:55.319601 (+2199838us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b180003a120. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b318282c 00007fb9b05ad9dc 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05c8155 00007fb9b6ec8167 00007fb9b34fd807 00007fb9b350cf0f
0806 20:35:55.319737 (+ 136us) log.cc:842] Serialized 14 byte log entry
0806 20:35:55.944908 (+625171us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:55.944992 (+ 84us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:55.945043 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.945110 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:55.945227 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":532712,"prepare.queue_time_us":836224,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"spinlock_wait_cycles":32381824,"thread_start_us":941750,"threads_started":2,"wal-append.queue_time_us":91233}
I20260806 20:35:55.954510 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1525) took 2707 ms. Trace:
W20260806 20:35:55.955824 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 67168.1us
I20260806 20:35:55.958118 14626 rpcz_store.cc:276] 0806 20:35:53.247679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.886533 (+638854us) service_pool.cc:227] Handling call
0806 20:35:55.948644 (+2062111us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b5800004298. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593bce 000000000034aea9 00007fb9b83bfa30 00007fb9b84857b4 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:35:55.954449 (+ 5805us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19977728}
I20260806 20:35:55.959571 14521 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:55.961997 14471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s user 0.029s sys 0.053s
I20260806 20:35:55.947208 17285 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:55.967322 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:55.968974 14595 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:55.966032 17137 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.007s
I20260806 20:35:55.973601 17197 raft_consensus.cc:2726] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:55.975826 17341 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:55.949314 16974 raft_consensus.cc:2741] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:35:55.955121 15178 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 618) took 1253 ms. Trace:
I20260806 20:35:55.961222 16962 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:55.977602 17384 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:55.978122 17285 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:55.978724 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:55.963042 17187 consensus_queue.cc:1035] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260806 20:35:55.979437 14550 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:55.949093 17269 raft_consensus.cc:2726] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:35:55.997157 14554 raft_consensus.cc:1226] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:55.984288 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:55.995810 14628 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.005s sys 0.034s
W20260806 20:35:56.003170 14628 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.588s user 0.005s sys 0.035s
I20260806 20:35:55.982797 14596 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:55.999392 14547 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:56.006261 14442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.023s sys 0.053s
W20260806 20:35:56.006615 14442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.023s sys 0.053s
I20260806 20:35:56.011308 14554 raft_consensus.cc:3032] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:56.011758 14554 raft_consensus.cc:729] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:56.013015 14554 consensus_queue.cc:260] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.014798 14554 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:55.977228 15178 rpcz_store.cc:276] 0806 20:35:54.701504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.712257 (+ 10753us) service_pool.cc:227] Handling call
0806 20:35:54.788668 (+ 76411us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.822293 (+ 33625us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:54.902562 (+ 80269us) log.cc:842] Serialized 14 byte log entry
0806 20:35:55.914516 (+1011954us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:55.928730 (+ 14214us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:55.934592 (+ 5862us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.941363 (+ 6771us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:55.951599 (+ 10236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":936598,"prepare.queue_time_us":78344,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":893650,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":8695,"thread_start_us":98000,"threads_started":1,"wal-append.queue_time_us":160501}
I20260806 20:35:55.920851 17175 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.020849 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.030418 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.030952 14478 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:56.031756 14554 raft_consensus.cc:1261] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260806 20:35:56.022449 14518 connection.cc:232] RPC call timeout handler was delayed by -6.4306817629840225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
W20260806 20:35:56.024706 14443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.029s sys 0.054s
W20260806 20:35:56.032977 14518 connection.cc:232] RPC call timeout handler was delayed by -6.4108817629748955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
W20260806 20:35:56.033245 14443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.029s sys 0.054s
W20260806 20:35:56.033464 14518 connection.cc:232] RPC call timeout handler was delayed by -6.4063817629939876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20260806 20:35:56.034045 14518 connection.cc:232] RPC call timeout handler was delayed by -6.3739817629975732s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20260806 20:35:56.034402 14518 connection.cc:232] RPC call timeout handler was delayed by -6.2812817629892379s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
W20260806 20:35:56.034825 14518 connection.cc:232] RPC call timeout handler was delayed by -6.1678817629872356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1879s for a response
W20260806 20:35:56.035190 14518 connection.cc:232] RPC call timeout handler was delayed by -6.0994817629980389s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
I20260806 20:35:56.036911 14631 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:56.037138 14596 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:56.038043 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 73889.6us
W20260806 20:35:56.036350 14518 connection.cc:232] RPC call timeout handler was delayed by -6.0076817629742436s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
I20260806 20:35:56.039053 14521 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:56.037062 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:56.046312 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:56.049302 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:55.950754 17355 consensus_queue.cc:1046] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260806 20:35:56.041988 14546 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:56.049737 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:55.946046 17180 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.052191 14554 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:55.977766 17147 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.056736 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:56.004599 17375 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.632s user 0.007s sys 0.009s
I20260806 20:35:56.057216 14473 raft_consensus.cc:1226] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:56.057916 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.058756 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:56.056003 17181 consensus_queue.cc:1046] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260806 20:35:56.066792 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.066928 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:56.024559 17111 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.009s sys 0.075s
I20260806 20:35:56.160252 17187 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:56.171710 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1532) took 1943 ms (client timeout 1707 ms). Trace:
I20260806 20:35:56.172224 17187 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.163105 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1729) took 3042 ms. Trace:
I20260806 20:35:56.173508 17285 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.173395 14551 rpcz_store.cc:276] 0806 20:35:53.120297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.092422 (+972125us) service_pool.cc:227] Handling call
0806 20:35:54.108024 (+ 15602us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:55.372844 (+1264820us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:55.372873 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:55.385153 (+ 12280us) log.cc:842] Serialized 26 byte log entry
0806 20:35:55.436125 (+ 50972us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:55.436177 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.436268 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:56.162839 (+726571us) raft_consensus.cc:1671] finished
0806 20:35:56.162859 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:56.163028 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1752404,"raft.queue_time_us":1272337,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":5312,"thread_start_us":48840,"threads_started":1,"wal-append.queue_time_us":629797}
W20260806 20:35:56.115600 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 76613.8us
I20260806 20:35:56.175696 14554 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:56.176281 14554 raft_consensus.cc:595] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:56.176647 14554 raft_consensus.cc:603] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:35:56.162173 14518 connection.cc:232] RPC call timeout handler was delayed by -5.5405817629944067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2576s for a response
I20260806 20:35:56.067231 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:56.177670 14518 connection.cc:232] RPC call timeout handler was delayed by -5.1400817629764788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30210000000000004s for a response
W20260806 20:35:56.172168 14625 rpcz_store.cc:269] 0806 20:35:54.228349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.228471 (+ 122us) service_pool.cc:227] Handling call
0806 20:35:56.171498 (+1943027us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540006fed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:56.171637 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3638236928}
W20260806 20:35:56.178066 14518 connection.cc:232] RPC call timeout handler was delayed by -4.9024817629833706s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3285s for a response
I20260806 20:35:56.179917 14596 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:56.180411 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:56.062767 14550 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:56.166610 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:56.067764 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.182181 15183 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:56.183751 14474 raft_consensus.cc:2379] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:56.017205 17137 consensus_queue.cc:1035] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.099s
W20260806 20:35:56.193997 15151 leader_election.cc:334] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.549s (SENT)
W20260806 20:35:56.195055 15151 leader_election.cc:334] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.664s (SENT)
I20260806 20:35:56.195513 15151 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:55.966884 17367 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.159s
I20260806 20:35:55.945722 17393 consensus_queue.cc:237] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.165201 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:56.197845 15186 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
W20260806 20:35:56.178424 14518 connection.cc:232] RPC call timeout handler was delayed by -3.0238985209725797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4218s for a response
I20260806 20:35:56.025326 17368 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.188231 14477 raft_consensus.cc:2396] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:35:56.121479 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:56.199178 14546 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:35:56.115135 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:56.191643 14479 raft_consensus.cc:2396] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 20:35:56.195986 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.202147 14470 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:56.199986 14596 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:56.052731 17296 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:35:56.207931 15185 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 620) took 1389 ms (client timeout 1802 ms). Trace:
I20260806 20:35:56.227520 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:56.228343 14551 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:56.227279 15185 rpcz_store.cc:269] 0806 20:35:54.818795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.818946 (+ 151us) service_pool.cc:227] Handling call
0806 20:35:56.207774 (+1388828us) spinlock_profiling.cc:243] Waited 99.4 ms on lock 0x7b54003787d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:56.207878 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228644608}
W20260806 20:35:56.177070 17379 log.cc:924] Time spent T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20260806 20:35:56.203871 14478 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
W20260806 20:35:56.053277 17331 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.000s sys 0.023s
I20260806 20:35:56.217831 17394 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:56.065513 16910 consensus_queue.cc:237] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:56.185366 17111 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.014s sys 0.075s
I20260806 20:35:56.224820 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.224715 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.226428 14518 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:56.224030 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.327992 17181 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:56.162261 16962 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:35:56.329879 14473 raft_consensus.cc:2379] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:56.232595 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1607) took 3761 ms. Trace:
I20260806 20:35:56.121748 17355 consensus_queue.cc:1225] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Peer 92828dd6dbe74f29a04d4e702b502621 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 20:35:56.233688 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.328962 14469 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:35:56.332343 14442 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:56.334870 14474 raft_consensus.cc:2354] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:35:56.115620 17372 consensus_queue.cc:1035] T f38960b1dd9245a8ba85fea601b09e94 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.103s
I20260806 20:35:56.235906 15183 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:56.155432 17352 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:56.248299 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.236939 17224 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.252887 17229 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.221490 17368 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.253875 17180 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.227420 17197 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:56.255192 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:56.362219 14625 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:56.091425 17313 consensus_queue.cc:1046] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.158s
I20260806 20:35:56.256098 17231 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.208557 17205 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 20:35:56.238890 17331 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.916s user 0.000s sys 0.026s
I20260806 20:35:56.375409 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:56.376271 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.376273 14631 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:56.232993 17296 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:56.339460 17193 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:56.333079 14479 raft_consensus.cc:2379] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:56.233768 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.235070 14550 raft_consensus.cc:3032] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:56.389914 14550 raft_consensus.cc:729] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:56.167085 17395 consensus_queue.cc:237] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.165550 17332 consensus_queue.cc:237] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.325510 17394 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:56.365873 15153 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:56.197561 17375 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.345875 17339 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.227746 17344 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:56.343348 15178 raft_consensus.cc:1226] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 20:35:56.341883 17119 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.344559 17147 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.406066 17147 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.359716 17220 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.329905 17181 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.341497 15182 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:35:56.365857 17329 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:56.321270 17235 consensus_queue.cc:1035] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260806 20:35:56.390997 14550 consensus_queue.cc:260] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.371467 17353 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:56.425369 17302 raft_consensus.cc:3032] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:35:56.425442 17353 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:56.425983 17302 raft_consensus.cc:729] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:56.426101 17353 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:56.404652 17119 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.347378 17111 consensus_queue.cc:1035] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260806 20:35:56.335812 17273 consensus_queue.cc:1046] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260806 20:35:56.377023 17197 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.418111 14629 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:56.359979 17352 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.365204 17224 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.421864 17187 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.422451 17147 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.422726 17368 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.423337 17374 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.455592 17374 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:56.424685 17375 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.424917 17205 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:56.375483 17229 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.428907 14550 raft_consensus.cc:3037] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:56.399911 17344 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:56.400549 17394 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:56.388940 17296 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:56.426990 17302 consensus_queue.cc:260] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.380395 17231 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.418672 17175 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.418807 17220 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:56.461297 14632 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.895s user 0.017s sys 0.033s
W20260806 20:35:56.474424 14632 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s user 0.018s sys 0.033s
W20260806 20:35:56.462520 17258 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s user 0.014s sys 0.022s
I20260806 20:35:56.461983 17344 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:35:56.480880 17258 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.030s user 0.014s sys 0.022s
I20260806 20:35:56.479841 17269 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:56.480207 17302 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:35:56.483525 16820 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.009s sys 0.038s
I20260806 20:35:56.510416 17352 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.504278 17258 raft_consensus.cc:3032] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:56.515226 17258 raft_consensus.cc:729] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:35:56.516885 17256 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.900s user 0.000s sys 0.056s
W20260806 20:35:56.532405 17343 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.004s sys 0.011s
W20260806 20:35:56.535156 17343 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s user 0.004s sys 0.011s
W20260806 20:35:56.641326 14444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.127s user 0.019s sys 0.057s
W20260806 20:35:56.642382 14444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.129s user 0.020s sys 0.058s
W20260806 20:35:56.049273 14415 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 500956bd1a3846a3a7b16c51ddecbc44 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20260806 20:35:56.643424 14415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 594047us
I20260806 20:35:56.653420 14444 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:56.341126 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.330499 14471 rpcz_store.cc:276] 0806 20:35:52.471192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:53.111275 (+640083us) service_pool.cc:227] Handling call
0806 20:35:53.111632 (+ 357us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:53.112068 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:53.112100 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:55.391014 (+2278914us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b2c0006bf28. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:55.426963 (+ 35949us) log.cc:842] Serialized 52 byte log entry
0806 20:35:55.962763 (+535800us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:56.014726 (+ 51963us) log.cc:842] Serialized 14 byte log entry
0806 20:35:56.029538 (+ 14812us) log.cc:842] Serialized 14 byte log entry
0806 20:35:56.030087 (+ 549us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:56.030180 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:56.232346 (+202166us) raft_consensus.cc:1671] finished
0806 20:35:56.232372 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:56.232538 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":161613,"prepare.queue_time_us":50111,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":26191,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":15732,"spinlock_wait_cycles":74001024,"thread_start_us":535318,"threads_started":1,"wal-append.queue_time_us":688413}
I20260806 20:35:56.655761 14478 raft_consensus.cc:2379] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 3: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:56.330916 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.342962 14477 raft_consensus.cc:2379] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:56.657935 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.657888 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:56.666399 14547 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.005s sys 0.005s
I20260806 20:35:56.463501 14546 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
W20260806 20:35:56.469548 14552 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s user 0.014s sys 0.040s
W20260806 20:35:56.820576 14552 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.268s user 0.015s sys 0.040s
I20260806 20:35:56.821722 17185 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.822134 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1603) took 3670 ms. Trace:
I20260806 20:35:56.822836 17185 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.824960 17181 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:56.676811 14555 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.000s sys 0.016s
W20260806 20:35:56.826133 14555 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s user 0.000s sys 0.017s
I20260806 20:35:56.827538 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1629) took 1413 ms. Trace:
W20260806 20:35:56.685391 14630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.091s user 0.007s sys 0.010s
W20260806 20:35:56.685439 14598 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.326s user 0.015s sys 0.021s
I20260806 20:35:56.689342 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1550) took 1391 ms. Trace:
I20260806 20:35:56.689847 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:55.957031 15179 rpcz_store.cc:276] 0806 20:35:54.513229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.517016 (+ 3787us) service_pool.cc:227] Handling call
0806 20:35:54.517387 (+ 371us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:54.517781 (+ 394us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.517815 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:54.518332 (+ 517us) log.cc:842] Serialized 26 byte log entry
0806 20:35:54.797317 (+278985us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:54.858398 (+ 61081us) log.cc:842] Serialized 14 byte log entry
0806 20:35:55.857398 (+999000us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b1800327dc0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05424ca 00007fb9b9c23f99 00007fb9b0406990 00007fb9b05c8864 00007fb9b05c8155 00007fb9b34eb3d0 00007fb9b34fd576 00007fb9b350e0d7 00007fb9b350e058
0806 20:35:55.884438 (+ 27040us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:55.898160 (+ 13722us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:55.901301 (+ 3141us) raft_consensus.cc:1671] finished
0806 20:35:55.906731 (+ 5430us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:55.906979 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":398,"prepare.queue_time_us":37834,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":997787,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":7541,"spinlock_wait_cycles":93982976,"thread_start_us":252549,"threads_started":1,"wal-append.queue_time_us":300004,"wal-append.run_cpu_time_us":2790,"wal-append.run_wall_time_us":1144043}
W20260806 20:35:56.833745 17312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.089s user 0.010s sys 0.055s
I20260806 20:35:56.833976 14473 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:35:56.834352 17312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.156s user 0.010s sys 0.055s
I20260806 20:35:56.835290 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:56.835644 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:56.836903 14470 consensus_meta.cc:287] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.634s user 0.012s sys 0.008s
I20260806 20:35:56.837080 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:56.837316 14470 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 8.
I20260806 20:35:56.837860 14625 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
W20260806 20:35:56.838632 15151 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.000s sys 0.018s
W20260806 20:35:56.839112 15151 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.636s user 0.000s sys 0.018s
I20260806 20:35:56.839213 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:56.840909 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.843127 14473 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:56.844566 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.709323 15182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:35:56.716696 16962 raft_consensus.cc:686] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:56.718271 16974 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:56.516721 17103 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.783s user 0.006s sys 0.013s
W20260806 20:35:56.724598 17263 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.012s sys 0.003s
I20260806 20:35:56.726956 17224 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:56.527007 17256 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s user 0.000s sys 0.057s
I20260806 20:35:56.754601 17394 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.755677 14475 raft_consensus.cc:1226] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260806 20:35:56.563283 17354 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.006s sys 0.020s
I20260806 20:35:56.762316 17111 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:56.763339 17119 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.877058 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 365) took 2583 ms. Trace:
I20260806 20:35:56.819515 17333 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:56.833139 14551 consensus_meta.cc:287] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.604s user 0.003s sys 0.019s
I20260806 20:35:56.830825 14632 rpcz_store.cc:276] 0806 20:35:55.298177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:55.298306 (+ 129us) service_pool.cc:227] Handling call
0806 20:35:55.298709 (+ 403us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:55.444514 (+145805us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:55.444558 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:55.450092 (+ 5534us) log.cc:842] Serialized 26 byte log entry
0806 20:35:56.474949 (+1024857us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:56.475005 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:56.475112 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:56.688722 (+213610us) raft_consensus.cc:1671] finished
0806 20:35:56.688777 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:56.689092 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15193,"raft.queue_time_us":151421,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":26775,"thread_start_us":1032060,"threads_started":2,"wal-append.queue_time_us":1070353}
I20260806 20:35:56.838536 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:56.839634 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 637188us
I20260806 20:35:56.839892 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:35:56.503159 16820 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.862s user 0.009s sys 0.038s
I20260806 20:35:56.713917 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:56.890319 14631 raft_consensus.cc:2379] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:56.840975 14625 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:56.891114 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:56.891801 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:56.848834 14469 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:56.892529 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:56.892931 17394 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:56.893383 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.894260 14469 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:56.894284 17231 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.897665 14551 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:56.891142 15150 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:56.891271 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:56.899207 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:56.851904 14554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s user 0.012s sys 0.004s
W20260806 20:35:56.853965 17160 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.024s user 0.006s sys 0.012s
I20260806 20:35:56.855214 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.845223 17220 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:56.849251 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.857755 17258 consensus_queue.cc:260] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.900434 14551 raft_consensus.cc:2396] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260806 20:35:56.867192 14598 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.508s user 0.016s sys 0.022s
W20260806 20:35:56.866220 14547 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s user 0.006s sys 0.005s
I20260806 20:35:56.871006 17337 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.871277 17395 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:56.871809 17276 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:56.927511 17337 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:56.869787 14630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.276s user 0.007s sys 0.010s
I20260806 20:35:56.822759 14552 rpcz_store.cc:276] 0806 20:35:53.152133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.764648 (+1612515us) service_pool.cc:227] Handling call
0806 20:35:54.781005 (+ 16357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:55.439767 (+658762us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:55.439813 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:55.440352 (+ 539us) log.cc:842] Serialized 26 byte log entry
0806 20:35:56.821409 (+1381057us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:56.821546 (+ 137us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:56.821706 (+ 160us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:56.821785 (+ 79us) raft_consensus.cc:1671] finished
0806 20:35:56.821803 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:56.822018 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1051333,"raft.queue_time_us":664073,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":100650,"thread_start_us":1268599,"threads_started":1,"wal-append.queue_time_us":1095334}
W20260806 20:35:56.876015 17263 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s user 0.012s sys 0.004s
I20260806 20:35:56.876753 16974 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.872013 15182 raft_consensus.cc:595] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:56.870930 17197 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:56.884462 17103 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.151s user 0.006s sys 0.013s
I20260806 20:35:56.902711 17205 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:56.937525 17205 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:56.884577 14479 raft_consensus.cc:2354] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:56.818189 17229 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.884291 17235 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.887029 17155 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:56.885180 17354 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.192s user 0.007s sys 0.020s
I20260806 20:35:56.827950 14555 rpcz_store.cc:276] 0806 20:35:55.414049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:55.414212 (+ 163us) service_pool.cc:227] Handling call
0806 20:35:55.414620 (+ 408us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:55.903130 (+488510us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:55.903158 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:55.906730 (+ 3572us) log.cc:842] Serialized 26 byte log entry
0806 20:35:56.826925 (+920195us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:56.827037 (+ 112us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:56.827148 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:56.827238 (+ 90us) raft_consensus.cc:1671] finished
0806 20:35:56.827255 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:56.827475 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":488048,"raft.queue_time_us":488207,"spinlock_wait_cycles":22382976,"thread_start_us":919484,"threads_started":1,"wal-append.queue_time_us":674839}
I20260806 20:35:56.784688 17147 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:35:56.901136 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 248071us
I20260806 20:35:56.893414 14632 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:56.913688 14521 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54498 dropped due to backpressure. The service queue is full; it has 50 items.
I20260806 20:35:56.901484 14470 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:56.922302 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:56.922648 14625 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:35:56.935745 14520 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.004s sys 0.023s
W20260806 20:35:56.977383 17353 consensus_meta.cc:287] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.550s user 0.010s sys 0.011s
I20260806 20:35:56.922394 15187 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:35:56.905622 17111 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.925469 17376 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:56.910315 17333 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.986161 17376 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.987615 17333 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:56.913908 14441 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.802s (SENT) [suppressed 20 similar messages]
I20260806 20:35:56.926947 17378 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:56.885509 14476 rpcz_store.cc:276] 0806 20:35:54.293084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.293242 (+ 158us) service_pool.cc:227] Handling call
0806 20:35:54.293616 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:54.473812 (+180196us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:54.473868 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:54.473925 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:54.473969 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:56.875867 (+2401898us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400331dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:56.875953 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:56.876194 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":413843712}
I20260806 20:35:56.933534 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1549) took 1657 ms. Trace:
I20260806 20:35:56.936231 14631 raft_consensus.cc:1261] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:35:56.936849 15182 raft_consensus.cc:603] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:35:56.931066 14598 connection.cc:232] RPC call timeout handler was delayed by -1.1187487530114595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260806 20:35:56.933413 17258 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:56.923774 14471 raft_consensus.cc:1226] T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 20:35:56.945783 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:56.944615 17235 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:56.900507 16844 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:57.001636 16844 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.001732 16286 raft_consensus.cc:3032] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:57.002032 16844 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.001787 17355 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.003037 17355 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.988385 17376 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.005126 14629 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 6. Ops: []
I20260806 20:35:57.005703 17355 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.006227 17269 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:57.006815 17269 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:57.008216 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.001857 14625 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:57.009753 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.010612 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.011220 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.012773 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.013175 14475 raft_consensus.cc:1226] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 20:35:57.014608 14475 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:57.015743 14475 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: []
I20260806 20:35:57.017238 14633 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:57.018566 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:57.018730 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 79949.8us
I20260806 20:35:57.020617 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.011643 14625 raft_consensus.cc:2396] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 20:35:57.021603 15186 raft_consensus.cc:2354] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:35:57.023672 14473 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.024782 17111 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.026073 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:57.027122 14442 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:57.028215 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 49869.6us
I20260806 20:35:57.029011 14442 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:57.029845 14548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1612) took 2807 ms (client timeout 2458 ms). Trace:
I20260806 20:35:57.029969 14553 raft_consensus.cc:3032] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:57.030511 14553 raft_consensus.cc:729] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:57.030333 14548 rpcz_store.cc:269] 0806 20:35:54.222587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.545093 (+322506us) service_pool.cc:227] Handling call
0806 20:35:57.029438 (+2484345us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b540018ecd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:57.029630 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5501508480}
I20260806 20:35:56.942512 17395 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.031800 14553 consensus_queue.cc:260] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.033203 14553 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.034075 17395 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.034653 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.035260 14550 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:57.038420 14627 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:56.904333 16962 consensus_queue.cc:237] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:56.976533 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.006s sys 0.032s
I20260806 20:35:56.945199 16974 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.990571 14630 rpcz_store.cc:276] 0806 20:35:55.275810 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:55.276504 (+ 694us) service_pool.cc:227] Handling call
0806 20:35:55.276921 (+ 417us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:55.509896 (+232975us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:55.509951 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:55.510667 (+ 716us) log.cc:842] Serialized 26 byte log entry
0806 20:35:56.933062 (+1422395us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:56.933163 (+ 101us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:56.933265 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:56.933355 (+ 90us) raft_consensus.cc:1671] finished
0806 20:35:56.933365 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:56.933476 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34396,"raft.queue_time_us":162473,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":9934,"thread_start_us":1463881,"threads_started":2,"wal-append.queue_time_us":1012233}
I20260806 20:35:56.976794 17296 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:56.983289 14520 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.004s sys 0.023s
I20260806 20:35:56.983992 14547 raft_consensus.cc:3037] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Advancing to term 9
W20260806 20:35:56.984736 17329 consensus_meta.cc:287] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.564s user 0.007s sys 0.010s
I20260806 20:35:56.984905 17337 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:56.985308 15181 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:35:56.993518 17137 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:56.982281 17276 raft_consensus.cc:686] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:56.995118 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.997937 14479 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:57.002254 16286 raft_consensus.cc:729] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:57.026233 14624 raft_consensus.cc:3037] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:56.943326 17147 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:56.945431 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:57.040375 14633 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:56.909636 17303 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:57.022199 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.010613 14626 raft_consensus.cc:2379] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:56.943231 17155 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:57.046423 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.006s sys 0.032s
I20260806 20:35:57.000900 17353 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.053099 14476 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:35:57.000385 17185 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.044986 17395 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:57.066520 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.069609 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:56.928105 14519 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54508 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1748)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 376)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1643)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1642)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1644)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1645)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 371)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1647)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1648)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1650)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1651)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1749)
I20260806 20:35:57.069891 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.071959 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:57.064918 17333 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:57.079232 17372 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:57.081080 14477 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:57.081696 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 207730us
I20260806 20:35:57.083351 14628 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260806 20:35:57.084306 17302 consensus_meta.cc:287] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.541s user 0.007s sys 0.007s
I20260806 20:35:57.084700 17155 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:57.086253 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 140978us
I20260806 20:35:57.086617 17353 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.087085 14444 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:57.087483 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.088012 17395 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.088660 14476 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:57.090148 17296 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.090245 17395 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.091500 14595 leader_election.cc:398] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:57.091823 14595 leader_election.cc:401] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:57.092571 17185 raft_consensus.cc:2741] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 20:35:57.095147 14595 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.096128 14631 raft_consensus.cc:2379] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:57.096951 15180 raft_consensus.cc:2396] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:57.097664 15150 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.098317 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.098222 17372 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.100230 17177 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:57.100694 17177 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.101140 17177 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:57.101303 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.102407 14625 raft_consensus.cc:2354] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:35:57.102700 14479 raft_consensus.cc:1261] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:35:57.105291 14444 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.106568 17374 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:57.108407 14444 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:57.109303 17263 raft_consensus.cc:2741] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 20:35:57.109231 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.110962 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.113517 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.114877 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.112207 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.120401 14552 raft_consensus.cc:2379] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:57.121373 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.122231 14552 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:57.122938 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.123401 14477 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:57.123754 14552 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:57.124356 17353 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:57.125084 17353 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.125435 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.126235 14546 raft_consensus.cc:2379] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:57.127027 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.127269 17353 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.127797 14546 raft_consensus.cc:2354] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:57.125748 14474 raft_consensus.cc:1226] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:57.130627 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 493) took 1087 ms. Trace:
I20260806 20:35:57.130823 16286 consensus_queue.cc:260] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.132740 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.130988 15187 rpcz_store.cc:276] 0806 20:35:56.042949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:56.043140 (+ 191us) service_pool.cc:227] Handling call
0806 20:35:56.066082 (+ 22942us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:56.985147 (+919065us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.130339 (+145192us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b54004c27d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:57.130538 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1952622336}
I20260806 20:35:57.132761 16286 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.127683 17372 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.143864 14633 raft_consensus.cc:2396] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:35:57.144846 14547 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 9.
I20260806 20:35:57.077862 17111 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:57.145705 14469 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:57.149694 14550 raft_consensus.cc:2396] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 20:35:57.132841 17276 consensus_queue.cc:237] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.142516 17296 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6.
I20260806 20:35:57.154410 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.155243 14548 raft_consensus.cc:2379] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:35:57.155481 17426 log.cc:924] Time spent T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260806 20:35:57.156137 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.157063 14548 raft_consensus.cc:2354] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:57.157827 17185 raft_consensus.cc:2741] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 20:35:57.157984 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.159454 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.160566 17386 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:35:57.160754 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.161028 17386 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.161460 17386 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:35:57.161525 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.162912 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:57.163102 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 936665us
I20260806 20:35:57.163587 14552 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:57.164530 14629 raft_consensus.cc:1226] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:35:57.164683 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.165581 14552 raft_consensus.cc:2354] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:57.166321 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.166901 14552 raft_consensus.cc:2362] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 9. Re-sending same reply.
I20260806 20:35:57.167791 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.168666 14552 raft_consensus.cc:2379] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:57.169330 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.169955 14552 raft_consensus.cc:2379] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:57.170614 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.171262 14552 raft_consensus.cc:2354] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:57.160195 14548 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.054335 17137 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.164122 14518 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.173609 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.160808 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.180742 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.182679 15187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:35:57.183384 15187 raft_consensus.cc:595] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.183818 15187 raft_consensus.cc:603] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:57.184702 14546 raft_consensus.cc:2396] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 20:35:57.185598 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.142478 17302 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.186378 17302 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.187269 17111 raft_consensus.cc:686] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:57.188362 17111 consensus_queue.cc:237] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.175835 14475 raft_consensus.cc:1226] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:35:57.186733 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.181711 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.173942 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1637) took 1132 ms. Trace:
I20260806 20:35:57.197006 14550 raft_consensus.cc:2354] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7.
I20260806 20:35:57.196729 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.204937 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1778) took 1157 ms. Trace:
I20260806 20:35:57.205518 14474 rpcz_store.cc:276] 0806 20:35:56.038349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:56.913860 (+875511us) service_pool.cc:227] Handling call
0806 20:35:56.914385 (+ 525us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.128466 (+214081us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.195057 (+ 66591us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54003c5854. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:35:57.195301 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":485536512}
I20260806 20:35:57.207887 14624 raft_consensus.cc:1261] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260806 20:35:57.147187 17333 raft_consensus.cc:686] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:57.213294 14627 raft_consensus.cc:1226] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:57.210525 16974 consensus_queue.cc:1035] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 20:35:57.215168 17205 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.216712 17393 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.211547 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.232540 14629 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:57.217376 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.233739 14546 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 9.
I20260806 20:35:57.234014 17255 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:57.218987 14479 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:57.235862 17337 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.236516 17337 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.237493 17353 consensus_queue.cc:1035] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:35:57.198894 17329 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.238884 17205 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.239274 17329 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.241179 14474 raft_consensus.cc:1261] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260806 20:35:57.241654 14474 pending_rounds.cc:76] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 3
I20260806 20:35:57.241899 17329 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.244180 17386 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.245338 14631 raft_consensus.cc:2354] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 8. Current term is 9.
I20260806 20:35:57.245980 17137 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.221055 14551 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.246738 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.247571 14632 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:35:56.944744 14521 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54498 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1748)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 376)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1643)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1642)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1644)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1645)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 371)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1647)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1648)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1650)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1651)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1749)
I20260806 20:35:57.228621 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.219972 17269 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.196811 14554 rpcz_store.cc:276] 0806 20:35:56.014628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:56.197747 (+183119us) service_pool.cc:227] Handling call
0806 20:35:56.198136 (+ 389us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:56.207896 (+ 9760us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:56.207926 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:56.230146 (+ 22220us) log.cc:842] Serialized 26 byte log entry
0806 20:35:56.941929 (+711783us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:57.067416 (+125487us) log.cc:842] Serialized 14 byte log entry
0806 20:35:57.073002 (+ 5586us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.083042 (+ 10040us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:57.140535 (+ 57493us) raft_consensus.cc:1671] finished
0806 20:35:57.140557 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:57.147325 (+ 6768us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16961,"prepare.queue_time_us":143822,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":14693,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":28868,"thread_start_us":670715,"threads_started":2,"wal-append.queue_time_us":518840}
I20260806 20:35:57.238639 17337 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.242194 14474 pending_rounds.cc:97] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:35:57.296245 14474 raft_consensus.cc:2866] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:57.242962 17147 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.239323 17393 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:57.298373 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 65673.1us
I20260806 20:35:57.298642 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1639) took 1247 ms. Trace:
I20260806 20:35:57.299641 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.300679 14552 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:35:57.301961 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.302839 14552 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:57.249843 14548 raft_consensus.cc:2396] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 20:35:57.305681 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.307277 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.253320 14469 raft_consensus.cc:1261] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:57.241201 16844 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:57.259219 17410 log.cc:924] Time spent T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.691s user 0.003s sys 0.000s
I20260806 20:35:57.261519 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:57.264253 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.313481 14555 raft_consensus.cc:2396] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 20:35:57.265827 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1779) took 1227 ms. Trace:
I20260806 20:35:57.266126 14551 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:57.269737 17177 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.208249 14624 pending_rounds.cc:76] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 5
I20260806 20:35:57.314436 14624 pending_rounds.cc:97] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260806 20:35:57.314718 14624 raft_consensus.cc:2866] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:57.225917 14473 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:57.314939 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 364) took 1057 ms. Trace:
W20260806 20:35:57.273172 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 109191us
W20260806 20:35:57.276549 17344 consensus_meta.cc:287] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.590s user 0.010s sys 0.009s
I20260806 20:35:57.279856 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.248492 14595 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:35:57.295406 17160 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.299882 17393 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.201344 17302 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:57.241595 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 133318us
I20260806 20:35:57.308214 14548 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:57.303942 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.315285 14555 rpcz_store.cc:276] 0806 20:35:56.257819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.263899 (+1006080us) service_pool.cc:227] Handling call
0806 20:35:57.314877 (+ 50978us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:57.313923 14471 rpcz_store.cc:276] 0806 20:35:56.038690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:56.922771 (+884081us) service_pool.cc:227] Handling call
0806 20:35:56.923341 (+ 570us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:56.996921 (+ 73580us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.265529 (+268608us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54001dca20. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b3483730 00007fb9b34f6b42 00007fb9b34f2f26 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:35:57.265758 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":276243200}
I20260806 20:35:57.299383 14553 rpcz_store.cc:276] 0806 20:35:56.051427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:56.205104 (+153677us) service_pool.cc:227] Handling call
0806 20:35:57.029657 (+824553us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.297303 (+267646us) raft_consensus.cc:1520] Early marking committed up to index 1
0806 20:35:57.297348 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:57.297399 (+ 51us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:57.297454 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.297579 (+ 125us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:57.298409 (+ 830us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b540018ecd0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:57.298567 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":167,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":148,"spinlock_wait_cycles":1825769856}
I20260806 20:35:57.316561 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.318385 14550 raft_consensus.cc:2379] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:57.319451 14547 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:57.325101 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.341980 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.348973 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.350497 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1636) took 1336 ms. Trace:
I20260806 20:35:57.351097 14546 raft_consensus.cc:3032] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:57.351056 14549 rpcz_store.cc:276] 0806 20:35:56.013856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:56.051757 (+ 37901us) service_pool.cc:227] Handling call
0806 20:35:57.350001 (+1298244us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540018e2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:57.350202 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2615496448}
I20260806 20:35:57.354139 16844 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.336026 17160 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.356601 17177 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.357911 17160 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.358340 15180 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 20:35:57.354228 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1535) took 2596 ms. Trace:
I20260806 20:35:57.359129 14624 rpcz_store.cc:276] 0806 20:35:54.757452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:54.759426 (+ 1974us) service_pool.cc:227] Handling call
0806 20:35:54.759796 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:55.755020 (+995224us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b1800146ee0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b352e856 00007fb9b3480f7e 00007fb9b34ea367 00007fb9b34ebb4e 00007fb9b34f5c89 00007fb9b34f6556 00007fb9b34f2edf 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684
0806 20:35:57.024373 (+1269353us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b58006572b0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b346ef63 00007fb9b34d2aba 00007fb9b34ea3af 00007fb9b34ebb4e 00007fb9b34f5c89 00007fb9b34f6556 00007fb9b34f2edf 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684
0806 20:35:57.338610 (+314237us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.354126 (+ 15516us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2259835520}
I20260806 20:35:57.360913 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:35:57.363273 17341 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.361397 14551 raft_consensus.cc:2396] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:35:57.364071 17341 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.366114 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.367354 14555 raft_consensus.cc:2396] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:35:57.297438 17333 consensus_queue.cc:237] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.351745 14546 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:57.373936 17393 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:57.374734 17393 raft_consensus.cc:686] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:57.376696 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:57.377084 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 47520.1us
I20260806 20:35:57.377449 15187 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:57.379642 14518 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.382313 16962 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:57.384372 14518 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:57.339125 17344 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.384781 14552 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 3.
W20260806 20:35:57.385360 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 44968.1us
I20260806 20:35:57.394270 14444 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.384895 14372 catalog_manager.cc:5495] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 2 to 4, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 4 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
W20260806 20:35:57.386207 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 137871us
I20260806 20:35:57.386379 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.387101 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.397017 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:35:57.389976 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47035.8us
I20260806 20:35:57.403939 14551 raft_consensus.cc:1226] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:35:57.405282 14547 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:35:57.406028 14547 raft_consensus.cc:595] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.406401 14547 raft_consensus.cc:603] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:35:57.409977 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 83041.2us
I20260806 20:35:57.404279 17255 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:35:57.413630 14554 raft_consensus.cc:2396] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 20:35:57.413733 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 367) took 1154 ms. Trace:
I20260806 20:35:57.414292 14551 rpcz_store.cc:276] 0806 20:35:56.259303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.364514 (+1105211us) service_pool.cc:227] Handling call
0806 20:35:57.393471 (+ 28957us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:57.412528 (+ 19057us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.413327 (+ 799us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b54002751d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:57.413608 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65386368}
I20260806 20:35:57.416847 14444 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.417487 14372 catalog_manager.cc:5495] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: config changed from index 3 to 4, term changed from 1 to 2, leader changed from 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129) to <none>, VOTER df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) evicted. New cstate: current_term: 2 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } } }
I20260806 20:35:57.419364 17353 raft_consensus.cc:2726] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:57.419415 17403 raft_consensus.cc:2726] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:57.418596 14444 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.421020 17403 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:57.421373 14548 raft_consensus.cc:3032] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:57.421445 17403 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.422052 14548 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:57.422173 17403 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.423269 14548 consensus_queue.cc:260] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.425235 14626 raft_consensus.cc:2379] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:57.425462 14550 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
I20260806 20:35:57.427000 14550 raft_consensus.cc:3032] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:57.427419 14550 raft_consensus.cc:729] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:57.428117 14554 raft_consensus.cc:1261] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:35:57.428825 14444 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.404757 14630 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:35:57.439172 14551 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:57.428350 14550 consensus_queue.cc:260] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.420141 14372 catalog_manager.cc:5495] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:57.441865 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.444298 16962 raft_consensus.cc:2726] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:35:57.445250 14369 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45970 (request call id 234) took 15098 ms (client timeout 15000 ms). Trace:
I20260806 20:35:57.445605 17341 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.449934 17344 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.437471 17185 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:35:57.440650 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1645) took 1056 ms. Trace:
I20260806 20:35:57.454257 14554 rpcz_store.cc:276] 0806 20:35:56.373304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.415640 (+1042336us) service_pool.cc:227] Handling call
0806 20:35:57.427780 (+ 12140us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.429688 (+ 1908us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.429979 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:57.457384 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.459250 14442 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.460309 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 371) took 1057 ms. Trace:
I20260806 20:35:57.460788 14551 rpcz_store.cc:276] 0806 20:35:56.395772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.428805 (+1033033us) service_pool.cc:227] Handling call
0806 20:35:57.429198 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.452647 (+ 23449us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.452934 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:57.375986 17393 consensus_queue.cc:237] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.455468 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.483035 14548 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.499825 15187 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:57.456244 17235 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:57.500734 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 67535.6us
I20260806 20:35:57.495895 14554 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:57.469589 17185 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:35:57.510790 14549 raft_consensus.cc:1261] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:35:57.515792 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.516110 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.516640 15187 raft_consensus.cc:2354] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7.
I20260806 20:35:57.520511 14554 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.522264 14553 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:57.523614 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.524437 14547 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:57.528010 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.528822 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:57.531476 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1745) took 1333 ms. Trace:
I20260806 20:35:57.536880 14549 rpcz_store.cc:276] 0806 20:35:56.180791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.369257 (+1188466us) service_pool.cc:227] Handling call
0806 20:35:57.481909 (+112652us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.513970 (+ 32061us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.514230 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:57.538704 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 369) took 1278 ms. Trace:
I20260806 20:35:57.540305 14551 raft_consensus.cc:3032] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:57.553690 14551 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:57.554654 14551 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.540985 14550 raft_consensus.cc:3037] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:57.555645 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 171457us
I20260806 20:35:57.521780 14546 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.548641 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:57.550132 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.556870 14547 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:57.560348 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:57.560550 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.561007 14471 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.561771 15187 raft_consensus.cc:2396] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:35:57.562557 17332 consensus_queue.cc:1035] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.028s
I20260806 20:35:57.567175 14444 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.547597 17341 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.568315 17341 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.570303 17341 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.570755 17341 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.571233 17341 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.544116 17160 consensus_queue.cc:1046] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 20:35:57.573235 17386 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.573496 17160 raft_consensus.cc:3032] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:57.573872 17160 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:57.574766 17160 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.553320 17137 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:57.577777 17269 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:57.445784 14369 rpcz_store.cc:269] 0806 20:35:42.305616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:42.316610 (+ 10994us) service_pool.cc:227] Handling call
0806 20:35:57.403392 (+15086782us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:35:49.197462 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:35:49.198142 (+ 680us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:35:49.198197 (+ 55us) write_op.cc:435] Acquired schema lock
0806 20:35:49.198231 (+ 34us) tablet.cc:660] Decoding operations
0806 20:35:49.414349 (+216118us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:35:49.452738 (+ 38389us) write_op.cc:641] Partition lock acquired for write op
0806 20:35:49.457626 (+ 4888us) tablet.cc:683] Acquiring locks for 17 operations
0806 20:35:50.019598 (+561972us) tablet.cc:699] Row locks acquired
0806 20:35:50.035208 (+ 15610us) write_op.cc:260] PREPARE: finished
0806 20:35:50.064807 (+ 29599us) write_op.cc:270] Start()
0806 20:35:50.120219 (+ 55412us) write_op.cc:276] Timestamp: P: 1786048550050794 usec, L: 0
0806 20:35:50.142448 (+ 22229us) op_driver.cc:331] REPLICATION: starting
0806 20:35:50.282418 (+139970us) log.cc:842] Serialized 6791 byte log entry
0806 20:35:50.774735 (+492317us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b05ad5fe 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb
0806 20:35:51.899884 (+1125149us) op_driver.cc:444] REPLICATION: finished
0806 20:35:52.149116 (+249232us) write_op.cc:301] APPLY: starting
0806 20:35:52.149257 (+ 141us) tablet.cc:1365] starting BulkCheckPresence
0806 20:35:52.177793 (+ 28536us) tablet.cc:1368] finished BulkCheckPresence
0806 20:35:52.177820 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:35:52.306060 (+128240us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:35:52.306486 (+ 426us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0806 20:35:52.306539 (+ 53us) write_op.cc:312] APPLY: finished
0806 20:35:52.307065 (+ 526us) log.cc:842] Serialized 119 byte log entry
0806 20:35:52.307283 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:52.307474 (+ 191us) write_op.cc:454] Released schema lock
0806 20:35:52.307545 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0806 20:35:52.307832 (+ 287us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:35:52.307856 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1720,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":222713,"apply.run_cpu_time_us":6407,"apply.run_wall_time_us":167859,"num_ops":17,"prepare.queue_time_us":496202,"prepare.run_cpu_time_us":91500,"prepare.run_wall_time_us":2116575,"raft.queue_time_us":431369,"raft.run_cpu_time_us":30500,"raft.run_wall_time_us":138684,"replication_time_us":1757325,"spinlock_wait_cycles":25326336,"thread_start_us":1832482,"threads_started":4,"wal-append.queue_time_us":1014616,"wal-append.run_cpu_time_us":45502,"wal-append.run_wall_time_us":2112191}]]}
I20260806 20:35:57.581468 14595 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
W20260806 20:35:57.572516 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 171918us
I20260806 20:35:57.587795 14470 raft_consensus.cc:1226] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:57.590736 14630 raft_consensus.cc:1261] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:57.592062 14553 rpcz_store.cc:276] 0806 20:35:56.260404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.403018 (+1142614us) service_pool.cc:227] Handling call
0806 20:35:57.514488 (+111470us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.522755 (+ 8267us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.538463 (+ 15708us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b54002d9bd0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:57.538632 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96702336}
W20260806 20:35:57.558562 17440 log.cc:924] Time spent T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20260806 20:35:57.576269 17137 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:57.594249 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:57.601032 14473 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:35:57.591563 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.604152 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.608469 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.610584 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.613883 14630 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:57.613092 14625 raft_consensus.cc:1261] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:57.601984 14518 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.618002 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.619545 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.622900 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.618867 14633 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:57.629724 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.631676 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.633092 14628 raft_consensus.cc:1226] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:35:57.634784 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.603513 17137 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:57.635317 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 54017.5us
I20260806 20:35:57.597743 17263 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:57.638473 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.641530 14630 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:57.609833 17155 consensus_queue.cc:1035] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
I20260806 20:35:57.645217 14631 raft_consensus.cc:2379] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:57.611327 17376 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:57.645869 14547 raft_consensus.cc:2379] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:35:57.646333 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.648276 15187 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:57.649086 14629 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:57.650501 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.651312 14553 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:35:57.622871 15182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:35:57.599578 16974 consensus_queue.cc:1035] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 20:35:57.665063 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.667420 14547 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:57.668825 17331 consensus_queue.cc:1046] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20260806 20:35:57.669903 17352 consensus_queue.cc:1035] T a94bd288a57145e8b254611b1dd82586 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260806 20:35:57.672801 17341 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.673700 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1644) took 1300 ms. Trace:
I20260806 20:35:57.664770 14549 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:57.685781 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.686970 14553 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:57.689749 14551 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.692960 14471 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:57.695534 14546 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.697185 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:57.691210 15182 raft_consensus.cc:595] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.700819 17332 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.700992 15182 raft_consensus.cc:603] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:35:57.701373 17332 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.702121 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.702628 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:35:57.703529 17332 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.708132 17177 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:57.708932 17177 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.711440 17177 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.667251 17403 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.714975 17403 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.717418 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:35:57.717816 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:57.718499 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:35:57.724617 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 142251us
I20260806 20:35:57.725823 14518 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:57.727767 14518 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.667569 17160 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.736574 17332 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.737242 17332 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.738034 17386 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:57.738606 17386 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.739292 17332 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.739698 17296 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:57.740520 17296 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.740984 17386 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:57.743265 17296 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:57.743934 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1747) took 1328 ms. Trace:
I20260806 20:35:57.744434 14553 rpcz_store.cc:276] 0806 20:35:56.415578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.696988 (+1281410us) service_pool.cc:227] Handling call
0806 20:35:57.743264 (+ 46276us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.743539 (+ 275us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:57.743563 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:57.743633 (+ 70us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:57.743676 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.743737 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:57.743859 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:57.684058 17376 raft_consensus.cc:686] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:57.765650 14547 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:57.674085 14548 rpcz_store.cc:276] 0806 20:35:56.372960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.396148 (+1023188us) service_pool.cc:227] Handling call
0806 20:35:57.421175 (+ 25027us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.673342 (+252167us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:57.673379 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:57.673418 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:57.673456 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.673516 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:57.673627 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17263,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":7243}
W20260806 20:35:57.774049 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 322768us
I20260806 20:35:57.774173 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.774529 17241 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:57.774887 14596 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:57.775595 14624 raft_consensus.cc:1226] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:57.775921 17255 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:57.776490 17255 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.776800 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.776952 14596 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.777637 14627 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:57.778295 14596 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.778522 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.778705 17341 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:35:57.779210 14596 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.779589 17332 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:57.765009 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:35:57.780118 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 52511.2us
I20260806 20:35:57.780445 15181 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.780783 17386 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:57.781196 17386 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.781780 17386 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:57.779366 17341 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:57.783264 17341 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.783725 17341 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.778524 17255 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:57.780184 17332 raft_consensus.cc:686] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:57.781184 14518 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:57.787075 15153 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.774963 17241 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:57.790015 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.790822 15187 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:35:57.792142 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:57.785993 17255 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:57.795266 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 368) took 1535 ms. Trace:
I20260806 20:35:57.797024 14626 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:57.796896 17376 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.795634 14555 rpcz_store.cc:276] 0806 20:35:56.259783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.373588 (+1113805us) service_pool.cc:227] Handling call
0806 20:35:57.374036 (+ 448us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.374374 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:57.374725 (+ 351us) log.cc:842] Serialized 14 byte log entry
0806 20:35:57.794957 (+420232us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:57.795012 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:57.795056 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.795115 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:57.795215 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25688,"prepare.queue_time_us":2140,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"thread_start_us":385578,"threads_started":1}
I20260806 20:35:57.802511 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1646) took 1412 ms. Trace:
I20260806 20:35:57.802098 17332 consensus_queue.cc:237] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:57.802927 14550 rpcz_store.cc:276] 0806 20:35:56.390512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.426335 (+1035823us) service_pool.cc:227] Handling call
0806 20:35:57.426826 (+ 491us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.802149 (+375323us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:35:57.802187 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:57.802230 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:57.802273 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.802339 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:57.802455 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36049}
I20260806 20:35:57.805001 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:57.767962 14372 catalog_manager.cc:5495] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 2 to 5, leader changed from 148bf2a11dba423ab364371f0458e871 (127.12.248.132) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 5 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:35:57.810989 14630 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:57.789286 17241 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:57.834574 14550 raft_consensus.cc:3032] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:35:57.835144 14550 raft_consensus.cc:729] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:57.841296 14596 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.837515 15153 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:57.845263 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.846240 15187 raft_consensus.cc:2379] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 8: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:35:57.809554 17177 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:57.855037 17258 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:35:57.864966 14550 consensus_queue.cc:260] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:57.867292 17302 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.877075 16974 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:57.880857 16974 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.881279 16974 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:57.886003 14555 raft_consensus.cc:1261] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:35:57.886713 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1749) took 1030 ms. Trace:
I20260806 20:35:57.887077 14555 rpcz_store.cc:276] 0806 20:35:56.856734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.806859 (+950125us) service_pool.cc:227] Handling call
0806 20:35:57.848552 (+ 41693us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.886413 (+ 37861us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.886636 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:57.867174 17177 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:57.895725 14555 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:57.898465 14596 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:57.910943 15187 raft_consensus.cc:1226] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 20:35:57.901298 17177 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:35:57.941800 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 344272us
I20260806 20:35:57.952332 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:35:57.909642 16962 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:57.889142 17302 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:58.027114 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1642) took 1963 ms. Trace:
I20260806 20:35:58.029533 14442 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:57.933168 17333 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:57.932986 17344 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:58.027949 14546 rpcz_store.cc:276] 0806 20:35:56.063871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.350360 (+1286489us) service_pool.cc:227] Handling call
0806 20:35:57.350873 (+ 513us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:58.026661 (+675788us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.026697 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:58.026741 (+ 44us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:58.026790 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:58.026857 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:58.027032 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:58.111841 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.112897 14469 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:58.114698 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.105365 14372 catalog_manager.cc:5495] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:58.122638 14624 raft_consensus.cc:1261] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:35:57.930819 17296 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.111s
I20260806 20:35:58.120575 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:58.135069 14628 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:58.131408 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:57.970705 15181 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:57.968547 17263 raft_consensus.cc:686] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:58.147831 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 118423us
I20260806 20:35:58.150552 14546 raft_consensus.cc:1261] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260806 20:35:58.091809 17344 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:58.156222 17344 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:58.161821 17276 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:58.143452 17333 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:58.162873 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.164613 17333 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:58.165345 15180 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:35:58.165549 14550 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:58.170105 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:58.170480 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:58.171159 14475 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:35:58.171236 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:58.172499 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.172809 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:58.173318 17447 log.cc:924] Time spent T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260806 20:35:58.173715 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.174165 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.174299 14477 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:35:58.170696 14479 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:58.174851 14470 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 4.
W20260806 20:35:58.130518 17393 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.072s sys 0.111s
I20260806 20:35:58.177637 14444 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:58.125576 17258 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:58.189670 14475 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 9.
I20260806 20:35:58.181850 14372 catalog_manager.cc:5495] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 3 to 5, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 5 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:58.177214 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.206444 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1746) took 1924 ms. Trace:
I20260806 20:35:58.223407 14552 rpcz_store.cc:276] 0806 20:35:56.282423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.395877 (+1113454us) service_pool.cc:227] Handling call
0806 20:35:57.453059 (+ 57182us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.497245 (+ 44186us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:57.542028 (+ 44783us) log.cc:842] Serialized 14 byte log entry
0806 20:35:57.901073 (+359045us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:57.901130 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:57.901180 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:57.901251 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:58.206181 (+304930us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b5400355f50. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:58.206363 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":39192,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"spinlock_wait_cycles":42806656,"thread_start_us":334066,"threads_started":1}
W20260806 20:35:58.209051 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 65317.8us
I20260806 20:35:58.213084 14478 raft_consensus.cc:2354] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:58.208498 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:58.183465 17393 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.073s sys 0.111s
I20260806 20:35:58.178464 17403 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:35:58.235373 14555 raft_consensus.cc:1261] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:35:58.258580 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:35:58.272174 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 61885.8us
W20260806 20:35:58.232980 17137 consensus_meta.cc:287] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.549s user 0.011s sys 0.001s
W20260806 20:35:58.286736 14554 consensus_meta.cc:287] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.760s user 0.010s sys 0.002s
I20260806 20:35:58.213080 14442 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:58.194911 17269 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:58.309993 14519 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.007s sys 0.053s
W20260806 20:35:58.310503 14519 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.007s sys 0.053s
I20260806 20:35:58.301275 14554 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:58.317459 14554 pending_rounds.cc:76] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 5
W20260806 20:35:58.322345 14551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.019s sys 0.046s
I20260806 20:35:58.344532 14554 pending_rounds.cc:97] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20260806 20:35:58.344815 14551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.019s sys 0.046s
I20260806 20:35:58.250761 17403 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:58.272519 17276 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:35:58.329584 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 116579us
W20260806 20:35:58.378768 14549 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.026s sys 0.040s
I20260806 20:35:58.380635 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.345126 14554 raft_consensus.cc:2866] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:35:58.333479 17111 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.018s sys 0.054s
W20260806 20:35:58.293184 17331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.031s sys 0.037s
W20260806 20:35:58.382859 14549 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.026s sys 0.040s
I20260806 20:35:58.383584 14475 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:58.346468 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1647) took 1945 ms. Trace:
I20260806 20:35:58.213536 17258 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:58.336935 14372 catalog_manager.cc:5495] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 2, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 2 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:58.392768 14551 rpcz_store.cc:276] 0806 20:35:56.401283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.461684 (+1060401us) service_pool.cc:227] Handling call
0806 20:35:57.513357 (+ 51673us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.665561 (+152204us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b5800728fb0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b346ef63 00007fb9b34d2aba 00007fb9b34ea3af 00007fb9b34ebb4e 00007fb9b34f5c89 00007fb9b34f6556 00007fb9b34f2edf 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684
0806 20:35:58.346016 (+680455us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.346058 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:58.346111 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:58.346176 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:58.346257 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:58.346401 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33041,"raft.queue_time_us":88827,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":19962,"spinlock_wait_cycles":76699904,"thread_start_us":550759,"threads_started":1}
I20260806 20:35:58.322083 17333 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260806 20:35:58.409008 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:58.412608 17155 consensus_queue.cc:1035] T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260806 20:35:58.415678 17386 consensus_meta.cc:287] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.633s user 0.008s sys 0.002s
I20260806 20:35:58.323066 17185 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:58.448191 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.031s sys 0.048s
W20260806 20:35:58.448683 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.031s sys 0.048s
I20260806 20:35:58.335340 17296 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:58.455350 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:58.321108 17255 raft_consensus.cc:2741] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
W20260806 20:35:58.400424 17331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.031s sys 0.038s
I20260806 20:35:58.386353 17276 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:58.399169 17111 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s user 0.018s sys 0.055s
W20260806 20:35:58.489317 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 101160us
I20260806 20:35:58.500286 14442 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:58.399631 17403 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:35:58.416153 17386 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:58.333237 17269 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:58.324681 17137 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:58.518224 14372 catalog_manager.cc:5495] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 2 to 3, leader changed from 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 3 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:58.333895 17312 consensus_queue.cc:1035] T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.166s
W20260806 20:35:58.529014 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 747947us
I20260806 20:35:58.523566 14628 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:35:58.499653 17276 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:58.565802 17258 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:58.565223 14372 catalog_manager.cc:5495] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 6, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130). New cstate: current_term: 6 leader_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } }
I20260806 20:35:58.566556 17258 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:58.570695 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1650) took 1897 ms. Trace:
I20260806 20:35:58.491570 17455 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:35:58.573836 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:58.581140 14551 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Advancing to term 9
W20260806 20:35:58.582810 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 82599us
I20260806 20:35:58.567215 14372 catalog_manager.cc:5495] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 reported cstate change: term changed from 1 to 4, leader changed from df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131) to 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129). New cstate: current_term: 4 leader_uuid: "92828dd6dbe74f29a04d4e702b502621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } health_report { overall_health: UNKNOWN } } }
I20260806 20:35:58.571146 14553 rpcz_store.cc:276] 0806 20:35:56.673434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.745882 (+1072448us) service_pool.cc:227] Handling call
0806 20:35:57.746489 (+ 607us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:57.772872 (+ 26383us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:57.772904 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:57.773454 (+ 550us) log.cc:842] Serialized 26 byte log entry
0806 20:35:58.190744 (+417290us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:58.190798 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:58.190882 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:58.570436 (+379554us) raft_consensus.cc:1671] finished
0806 20:35:58.570460 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:58.570629 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28770,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":416859,"threads_started":1,"wal-append.queue_time_us":794964}
I20260806 20:35:58.588634 17455 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:58.606952 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:58.609306 15153 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.015s sys 0.016s
I20260806 20:35:58.604703 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1666) took 1028 ms. Trace:
I20260806 20:35:58.609915 14546 rpcz_store.cc:276] 0806 20:35:57.575807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.458745 (+882938us) service_pool.cc:227] Handling call
0806 20:35:58.500148 (+ 41403us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:58.592968 (+ 92820us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.593019 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:58.604307 (+ 11288us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:58.604372 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:58.604467 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:58.604621 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:35:58.494827 17185 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:58.500548 17296 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:58.378304 17263 consensus_queue.cc:237] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:58.627907 14551 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 9.
W20260806 20:35:58.623235 15153 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.015s sys 0.016s
W20260806 20:35:58.636075 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 780881us
W20260806 20:35:58.623739 14521 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.003s sys 0.028s
W20260806 20:35:58.638643 14521 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s user 0.003s sys 0.028s
I20260806 20:35:58.612314 14518 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:58.649825 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:58.591980 17331 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:35:58.630860 17303 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.024s sys 0.032s
W20260806 20:35:58.653854 17303 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.774s user 0.024s sys 0.032s
I20260806 20:35:58.605098 17302 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:58.662278 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:58.656850 14442 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:58.567219 17258 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:58.612179 17455 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:58.659281 14546 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
W20260806 20:35:58.710799 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 54035.1us
I20260806 20:35:58.669207 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:58.719173 14551 raft_consensus.cc:3037] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:35:58.720542 14548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.013s sys 0.022s
W20260806 20:35:58.721081 14548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.831s user 0.014s sys 0.022s
W20260806 20:35:58.671667 14598 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.003s sys 0.028s
W20260806 20:35:58.722097 14598 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.004s sys 0.028s
W20260806 20:35:58.675191 14520 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.006s sys 0.055s
W20260806 20:35:58.722654 14520 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.884s user 0.006s sys 0.055s
W20260806 20:35:58.722688 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.017s sys 0.015s
W20260806 20:35:58.723338 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.017s sys 0.015s
W20260806 20:35:58.725827 14473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.007s sys 0.031s
W20260806 20:35:58.722180 14547 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.008s sys 0.027s
W20260806 20:35:58.726439 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 66060.7us
I20260806 20:35:58.652791 17331 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:35:58.726668 14547 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.888s user 0.008s sys 0.028s
W20260806 20:35:58.726248 14473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.008s sys 0.031s
I20260806 20:35:58.732148 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:58.734917 14546 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:35:58.737124 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:58.722115 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 385) took 2046 ms. Trace:
W20260806 20:35:58.726486 14631 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.013s sys 0.026s
W20260806 20:35:58.747365 14631 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.013s sys 0.026s
I20260806 20:35:58.747726 14553 raft_consensus.cc:2396] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 20:35:58.722599 17376 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.009s sys 0.016s
I20260806 20:35:58.750224 14548 rpcz_store.cc:276] 0806 20:35:56.675427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.771962 (+1096535us) service_pool.cc:227] Handling call
0806 20:35:57.772291 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:57.793182 (+ 20891us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:57.806539 (+ 13357us) log.cc:842] Serialized 14 byte log entry
0806 20:35:58.721666 (+915127us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:58.721741 (+ 75us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:58.721812 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:58.721894 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:58.722047 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13647,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"thread_start_us":831487,"threads_started":1}
I20260806 20:35:58.728374 17459 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:35:58.709193 17462 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:58.759804 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1648) took 2092 ms. Trace:
I20260806 20:35:58.738740 17331 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:58.797559 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.017s sys 0.017s
I20260806 20:35:58.788208 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:35:58.798425 14553 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:58.787900 14549 rpcz_store.cc:276] 0806 20:35:56.667354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.719166 (+1051812us) service_pool.cc:227] Handling call
0806 20:35:57.719607 (+ 441us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:57.719993 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:57.720033 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:57.751259 (+ 31226us) log.cc:842] Serialized 26 byte log entry
0806 20:35:58.391423 (+640164us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:58.391459 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:58.391527 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:58.759581 (+368054us) raft_consensus.cc:1671] finished
0806 20:35:58.759608 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:58.759741 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":637095,"threads_started":1,"wal-append.queue_time_us":913022}
W20260806 20:35:58.789016 14626 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.013s sys 0.026s
W20260806 20:35:58.799933 14626 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.675s user 0.013s sys 0.026s
W20260806 20:35:58.800470 14629 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.014s sys 0.030s
W20260806 20:35:58.798007 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.017s sys 0.017s
W20260806 20:35:58.801990 14629 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.014s sys 0.030s
I20260806 20:35:58.804353 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:35:58.832013 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 242574us
I20260806 20:35:58.803581 14551 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
W20260806 20:35:58.772014 17376 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.009s sys 0.016s
I20260806 20:35:58.924563 14548 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
I20260806 20:35:58.924988 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:58.927534 14471 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:35:58.928546 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:35:58.924867 14551 raft_consensus.cc:1261] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260806 20:35:58.862129 17177 consensus_meta.cc:287] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.871s user 0.012s sys 0.000s
I20260806 20:35:58.834913 17386 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:58.837088 17459 raft_consensus.cc:686] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:35:58.927066 17448 log.cc:924] Time spent T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.142s user 0.003s sys 0.009s
W20260806 20:35:58.966704 15151 leader_election.cc:334] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.077s (SENT)
W20260806 20:35:58.958884 14441 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.009s sys 0.033s
W20260806 20:35:58.979489 14441 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.643s user 0.010s sys 0.033s
W20260806 20:35:58.980010 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 656101us
I20260806 20:35:58.980729 14548 raft_consensus.cc:1261] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:35:58.970345 14518 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:58.859776 17480 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:58.970186 14470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.029s sys 0.082s
W20260806 20:35:58.985702 14470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s user 0.029s sys 0.082s
W20260806 20:35:58.958170 17450 log.cc:924] Time spent T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.101s user 0.000s sys 0.017s
I20260806 20:35:58.921381 17258 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:58.984551 14550 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.009s sys 0.022s
W20260806 20:35:59.004953 14550 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.010s sys 0.022s
I20260806 20:35:59.005826 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1756) took 1812 ms. Trace:
I20260806 20:35:59.006177 14550 rpcz_store.cc:276] 0806 20:35:57.193166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.804023 (+610857us) service_pool.cc:227] Handling call
0806 20:35:57.804332 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:59.005423 (+1201091us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:59.005476 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:59.005530 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:59.005584 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.005661 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.005773 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":627725,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":32246,"thread_start_us":742596,"threads_started":1}
I20260806 20:35:58.990485 14551 raft_consensus.cc:1261] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
W20260806 20:35:58.991470 14476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.011s sys 0.024s
I20260806 20:35:58.859972 17137 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:59.009871 14476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s user 0.011s sys 0.024s
I20260806 20:35:59.011286 14553 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
W20260806 20:35:59.012694 14474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.009s sys 0.044s
W20260806 20:35:59.013327 14474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.705s user 0.009s sys 0.045s
W20260806 20:35:58.999950 14627 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.820s user 0.013s sys 0.023s
I20260806 20:35:58.952309 17376 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:59.015712 14627 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.836s user 0.013s sys 0.023s
W20260806 20:35:58.976558 17147 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.194s user 0.018s sys 0.017s
W20260806 20:35:59.012634 15152 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.003s sys 0.021s
W20260806 20:35:59.035228 15152 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.858s user 0.003s sys 0.022s
I20260806 20:35:58.798115 17462 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:59.038530 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.014s sys 0.026s
I20260806 20:35:59.038846 14548 raft_consensus.cc:3032] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:59.040522 14548 raft_consensus.cc:729] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:35:59.029783 14471 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:35:59.043011 14551 pending_rounds.cc:76] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 3
I20260806 20:35:59.043315 14551 pending_rounds.cc:97] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:35:59.043562 14551 raft_consensus.cc:2866] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:35:59.026253 17147 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.244s user 0.018s sys 0.017s
I20260806 20:35:59.007370 17155 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:35:59.034409 17332 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.006s sys 0.031s
I20260806 20:35:58.978348 17177 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.037235 17468 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:59.034232 17137 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:59.048100 16974 consensus_meta.cc:287] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.166s user 0.009s sys 0.005s
W20260806 20:35:59.064972 14554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.016s sys 0.035s
W20260806 20:35:59.065344 14554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s user 0.016s sys 0.035s
W20260806 20:35:59.058950 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.010s sys 0.018s
W20260806 20:35:59.067070 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.889s user 0.010s sys 0.019s
W20260806 20:35:59.040031 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.850s user 0.015s sys 0.026s
W20260806 20:35:59.064213 17463 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.297s user 0.000s sys 0.001s
I20260806 20:35:59.042210 17269 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:35:59.055516 17332 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.801s user 0.006s sys 0.031s
W20260806 20:35:59.069813 14555 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.016s sys 0.036s
I20260806 20:35:59.071657 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:59.071869 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:59.094728 14479 consensus_meta.cc:287] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.864s user 0.012s sys 0.000s
W20260806 20:35:59.094641 14555 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.578s user 0.016s sys 0.037s
I20260806 20:35:59.095260 14479 raft_consensus.cc:2454] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:35:59.063651 14548 consensus_queue.cc:260] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:59.054769 17341 consensus_meta.cc:287] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.271s user 0.015s sys 0.000s
W20260806 20:35:59.084759 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 136021us
I20260806 20:35:59.057480 17185 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.097911 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:59.048659 17296 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:59.098448 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:35:59.101691 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 145974us
I20260806 20:35:59.033903 17376 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:59.169726 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 106752us
I20260806 20:35:59.115213 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:59.002202 17480 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.096846 17341 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.136101 14624 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:59.161924 17474 log.cc:924] Time spent T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260806 20:35:59.092706 17473 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:59.075210 17155 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.069679 17137 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.115113 15150 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:59.171155 14625 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:59.176019 14441 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:35:59.099745 17160 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 1.371s user 0.008s sys 0.006s
I20260806 20:35:59.197943 14518 leader_election.cc:398] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:35:59.198307 14518 leader_election.cc:401] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:59.162528 14548 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:59.174366 14469 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.035s sys 0.037s
W20260806 20:35:59.197846 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 82825.6us
W20260806 20:35:59.199620 14469 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s user 0.035s sys 0.037s
W20260806 20:35:59.182144 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 107965us
W20260806 20:35:59.202452 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 929266us
I20260806 20:35:59.185606 17296 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:59.203246 14444 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:59.203652 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1645) took 1092 ms. Trace:
I20260806 20:35:59.203583 17296 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.204389 15150 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:59.204607 15187 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:59.205322 17258 consensus_queue.cc:1046] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:35:59.206208 17296 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:59.208076 14521 leader_election.cc:398] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:35:59.210180 14521 leader_election.cc:401] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260806 20:35:59.208323 17137 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:35:59.211216 17453 log.cc:924] Time spent T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.410s user 0.000s sys 0.001s
I20260806 20:35:59.211328 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1656) took 1040 ms. Trace:
I20260806 20:35:59.211869 17462 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:59.208554 17473 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.211791 14470 rpcz_store.cc:276] 0806 20:35:58.170782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.175500 (+ 4718us) service_pool.cc:227] Handling call
0806 20:35:58.210769 (+ 35269us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:58.308121 (+ 97352us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.308154 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:58.309083 (+ 929us) log.cc:842] Serialized 52 byte log entry
0806 20:35:58.999903 (+690820us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:58.999953 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.000025 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:59.211088 (+211063us) raft_consensus.cc:1671] finished
0806 20:35:59.211111 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.211269 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":193192,"raft.queue_time_us":65496,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":80090,"thread_start_us":601021,"threads_started":1,"wal-append.queue_time_us":875324}
I20260806 20:35:59.209304 17160 raft_consensus.cc:3032] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:59.213883 17473 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:59.214079 17160 raft_consensus.cc:729] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:59.214794 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1647) took 1103 ms. Trace:
I20260806 20:35:59.215018 17160 consensus_queue.cc:260] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.216092 17341 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.215195 14476 rpcz_store.cc:276] 0806 20:35:58.111655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.130879 (+ 19224us) service_pool.cc:227] Handling call
0806 20:35:58.177921 (+ 47042us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:58.265019 (+ 87098us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.265069 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:58.384753 (+119684us) log.cc:842] Serialized 52 byte log entry
0806 20:35:59.010783 (+626030us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:59.010835 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.010922 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:59.214567 (+203645us) raft_consensus.cc:1671] finished
0806 20:35:59.214590 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.214730 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19804,"raft.queue_time_us":102718,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":26494,"thread_start_us":599930,"threads_started":1,"wal-append.queue_time_us":800963}
I20260806 20:35:59.216542 17472 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:59.204174 14473 rpcz_store.cc:276] 0806 20:35:58.110928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.111093 (+ 165us) service_pool.cc:227] Handling call
0806 20:35:58.111561 (+ 468us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:58.111807 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.111839 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:58.112368 (+ 529us) log.cc:842] Serialized 26 byte log entry
0806 20:35:58.734566 (+622198us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:35:58.734617 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:58.734698 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:59.203420 (+468722us) raft_consensus.cc:1671] finished
0806 20:35:59.203439 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.203572 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":621742,"threads_started":1,"wal-append.queue_time_us":924548}
I20260806 20:35:59.217219 17472 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.217705 15153 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:59.219762 17472 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.216323 17160 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:59.222298 17479 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.273s user 0.000s sys 0.000s
I20260806 20:35:59.223243 17497 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:35:59.206465 17258 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:59.225664 17258 raft_consensus.cc:686] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:35:59.226764 17258 consensus_queue.cc:237] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.102362 17459 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.208168 17155 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:59.202488 17302 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:59.229022 17302 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:59.229136 17341 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:59.229454 17302 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:59.229647 17341 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.231796 15153 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:59.233294 17303 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:35:59.233366 17480 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:59.234256 17263 raft_consensus.cc:2741] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:35:59.234503 14441 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:35:59.234791 17341 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:59.235503 14441 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:59.235642 17263 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:59.238843 14444 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:59.239665 17458 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:59.240619 17333 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:59.242255 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:59.242511 14444 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:59.245543 17177 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:35:59.247542 17344 consensus_meta.cc:287] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.090s user 0.010s sys 0.003s
I20260806 20:35:59.248883 17457 consensus_queue.cc:1046] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:35:59.250216 17457 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:59.250871 17457 raft_consensus.cc:3032] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:35:59.251256 17457 raft_consensus.cc:729] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:59.255303 17333 consensus_queue.cc:1046] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:35:59.257794 17312 raft_consensus.cc:3032] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:35:59.258256 17312 raft_consensus.cc:729] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:59.259279 17312 consensus_queue.cc:260] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.261343 17185 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:59.262081 17185 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.173359 17466 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:35:59.075599 16974 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.277336 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:59.236189 17263 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:59.298785 14444 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:35:59.300199 14632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1577) took 1672 ms (client timeout 1703 ms). Trace:
W20260806 20:35:59.305361 14632 rpcz_store.cc:269] 0806 20:35:57.603593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.603822 (+ 229us) service_pool.cc:227] Handling call
0806 20:35:59.276147 (+1672325us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540006efd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:59.276332 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3080682240}
W20260806 20:35:59.326807 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.050s sys 0.148s
I20260806 20:35:59.327044 14630 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:35:59.299957 17476 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.470s user 0.001s sys 0.000s
W20260806 20:35:59.327229 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.050s sys 0.148s
I20260806 20:35:59.248088 17344 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:59.329444 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 121540us
I20260806 20:35:59.335106 14521 leader_election.cc:398] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 8. Current term is 9.
I20260806 20:35:59.335479 14521 leader_election.cc:401] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260806 20:35:59.314782 17263 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:59.315625 17276 consensus_meta.cc:287] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.754s user 0.013s sys 0.000s
I20260806 20:35:59.298207 17466 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:35:59.315373 17241 consensus_meta.cc:287] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.446s user 0.010s sys 0.004s
I20260806 20:35:59.338792 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1585) took 1298 ms. Trace:
I20260806 20:35:59.329627 17312 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:59.358276 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 171830us
W20260806 20:35:59.346091 14442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.014s sys 0.048s
W20260806 20:35:59.367370 14442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.014s sys 0.048s
I20260806 20:35:59.369508 14628 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:35:59.362314 14595 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.374835 14548 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:35:59.361099 14624 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:35:59.337545 16962 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:59.338048 17462 raft_consensus.cc:2726] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:35:59.379915 14595 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.445562 14548 raft_consensus.cc:3032] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:35:59.252250 17457 consensus_queue.cc:260] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.339753 17486 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:59.449856 14548 raft_consensus.cc:729] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:35:59.363864 17456 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.531s user 0.007s sys 0.007s
I20260806 20:35:59.340226 17473 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:59.451886 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 72088.1us
I20260806 20:35:59.362071 14626 rpcz_store.cc:276] 0806 20:35:58.040035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.040181 (+ 146us) service_pool.cc:227] Handling call
0806 20:35:58.124190 (+ 84009us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:58.124444 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:58.838383 (+713939us) log.cc:842] Serialized 14 byte log entry
0806 20:35:59.338194 (+499811us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:59.338262 (+ 68us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:59.338502 (+ 240us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.338589 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.338735 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6606,"prepare.queue_time_us":64696,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":3273,"raft.queue_time_us":32422,"thread_start_us":1168283,"threads_started":2}
I20260806 20:35:59.380218 17137 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:59.360711 17467 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:59.462018 14595 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.362012 17466 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:35:59.460702 17403 consensus_meta.cc:287] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.890s user 0.012s sys 0.001s
I20260806 20:35:59.470710 14548 consensus_queue.cc:260] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.465739 14596 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:59.473292 14595 leader_election.cc:302] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.360157 17276 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.481272 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1587) took 1440 ms. Trace:
I20260806 20:35:59.366675 17241 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.460147 17296 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:35:59.462769 17462 raft_consensus.cc:2726] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 8. Current term is 9.
I20260806 20:35:59.490494 14595 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:35:59.469206 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 169885us
I20260806 20:35:59.446738 17393 consensus_queue.cc:1035] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260806 20:35:59.473165 17457 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:35:59.495815 14596 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:59.490116 17332 raft_consensus.cc:2726] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:35:59.508313 14475 raft_consensus.cc:2379] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:59.512816 14595 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:59.510300 14630 raft_consensus.cc:1261] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:35:59.450935 16962 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:59.512056 15187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 645) took 1570 ms (client timeout 1925 ms). Trace:
W20260806 20:35:59.521510 15187 rpcz_store.cc:269] 0806 20:35:57.935665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.951968 (+ 16303us) service_pool.cc:227] Handling call
0806 20:35:59.505795 (+1553827us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400710554. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:35:59.505922 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2877576576}
I20260806 20:35:59.512239 14552 raft_consensus.cc:1226] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 3. Current term is 4. Ops: []
I20260806 20:35:59.471875 17467 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.501586 17137 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.488698 17374 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 20:35:59.489305 17403 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.504359 17464 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
W20260806 20:35:59.515945 17492 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 20:35:59.528724 14596 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.466666 17473 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:59.530375 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 360526us
I20260806 20:35:59.533635 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:59.490231 14627 rpcz_store.cc:276] 0806 20:35:58.040745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.040842 (+ 97us) service_pool.cc:227] Handling call
0806 20:35:58.120018 (+ 79176us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:58.138982 (+ 18964us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:59.016180 (+877198us) log.cc:842] Serialized 14 byte log entry
0806 20:35:59.480837 (+464657us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:59.480901 (+ 64us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:59.480961 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.481059 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.481202 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":142083,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":7851,"thread_start_us":1266526,"threads_started":2}
I20260806 20:35:59.505373 17459 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:59.537860 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:59.539335 14548 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:59.548331 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:59.539388 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1665) took 1963 ms. Trace:
I20260806 20:35:59.491714 17185 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.460139 17475 consensus_queue.cc:1035] T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260806 20:35:59.543946 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:59.523042 17462 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:59.525343 17296 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:35:59.560370 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 225459us
I20260806 20:35:59.542073 17386 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:35:59.465022 17486 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.562961 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1650) took 1394 ms. Trace:
I20260806 20:35:59.549563 14555 rpcz_store.cc:276] 0806 20:35:57.575506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.342253 (+766747us) service_pool.cc:227] Handling call
0806 20:35:58.397640 (+ 55387us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:58.480764 (+ 83124us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:35:59.095588 (+614824us) log.cc:842] Serialized 14 byte log entry
0806 20:35:59.539060 (+443472us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:59.539113 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:59.539152 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.539211 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.539336 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":825930,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":3311,"raft.queue_time_us":67987,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":1016458,"threads_started":3}
I20260806 20:35:59.565852 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:59.567915 14473 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:35:59.580684 14470 raft_consensus.cc:2379] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:35:59.568778 14627 raft_consensus.cc:1261] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 20:35:59.539235 16974 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.537581 17505 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:35:59.561702 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:59.556356 17456 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:59.582425 14630 raft_consensus.cc:1261] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260806 20:35:59.544055 17465 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:59.583987 14476 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:35:59.588670 14630 pending_rounds.cc:76] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting all ops after (but not including) 2
I20260806 20:35:59.598425 14630 pending_rounds.cc:97] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 20:35:59.579008 14474 rpcz_store.cc:276] 0806 20:35:58.168343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.179343 (+ 11000us) service_pool.cc:227] Handling call
0806 20:35:58.209019 (+ 29676us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:58.209292 (+ 273us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 20:35:58.307714 (+ 98422us) log.cc:842] Serialized 14 byte log entry
0806 20:35:59.379470 (+1071756us) log.cc:842] Serialized 14 byte log entry
0806 20:35:59.556751 (+177281us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:59.561231 (+ 4480us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:35:59.561339 (+ 108us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.561519 (+ 180us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.562881 (+ 1362us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24488,"prepare.queue_time_us":608034,"prepare.run_cpu_time_us":223,"prepare.run_wall_time_us":3184,"raft.queue_time_us":202025,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":2856,"thread_start_us":1204704,"threads_started":3,"wal-append.queue_time_us":905589}
I20260806 20:35:59.598754 14630 raft_consensus.cc:2866] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:59.559791 17376 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:35:59.566570 17147 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:35:59.615831 14548 raft_consensus.cc:1261] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260806 20:35:59.562323 17459 raft_consensus.cc:686] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:59.618508 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:59.573215 17386 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:59.656785 14555 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 9.
I20260806 20:35:59.616429 14548 pending_rounds.cc:76] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 2
I20260806 20:35:59.663478 14548 pending_rounds.cc:97] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 20:35:59.663829 14548 raft_consensus.cc:2866] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:35:59.665520 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:59.656065 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:59.667976 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1668) took 2091 ms. Trace:
I20260806 20:35:59.577006 17464 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:35:59.669382 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 89834.2us
I20260806 20:35:59.671273 14521 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:59.668524 14549 rpcz_store.cc:276] 0806 20:35:57.576690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.799153 (+1222463us) service_pool.cc:227] Handling call
0806 20:35:58.863175 (+ 64022us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:58.863428 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:35:59.458525 (+595097us) log.cc:842] Serialized 14 byte log entry
0806 20:35:59.667647 (+209122us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:59.667715 (+ 68us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:59.667765 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.667821 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.667923 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46883,"prepare.queue_time_us":495333,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"thread_start_us":579652,"threads_started":2}
I20260806 20:35:59.576869 17344 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.597241 17296 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:35:59.672802 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 144188us
I20260806 20:35:59.597579 17235 raft_consensus.cc:3032] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:35:59.683977 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 97399.8us
I20260806 20:35:59.684197 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:35:59.674733 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:35:59.685833 14633 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:35:59.572700 17462 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.688156 14444 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:35:59.689513 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 103676us
I20260806 20:35:59.690843 14555 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:35:59.675773 17455 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.939s user 0.006s sys 0.006s
I20260806 20:35:59.669989 17386 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:35:59.675015 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 141482us
I20260806 20:35:59.687309 14596 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:59.692996 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1661) took 2148 ms. Trace:
I20260806 20:35:59.704818 14595 leader_election.cc:398] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
I20260806 20:35:59.704210 14554 rpcz_store.cc:276] 0806 20:35:57.544391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.316774 (+772383us) service_pool.cc:227] Handling call
0806 20:35:58.317228 (+ 454us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:58.452320 (+135092us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.452368 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:58.452976 (+ 608us) log.cc:842] Serialized 26 byte log entry
0806 20:35:59.066887 (+613911us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:59.066922 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.067004 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:35:59.692735 (+625731us) raft_consensus.cc:1671] finished
0806 20:35:59.692764 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:35:59.692922 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":539301,"threads_started":1,"wal-append.queue_time_us":829700}
I20260806 20:35:59.705255 14595 leader_election.cc:401] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:35:59.677147 17296 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:59.716825 15151 leader_election.cc:302] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.688637 17374 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:35:59.689139 17505 raft_consensus.cc:686] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:35:59.684314 17235 raft_consensus.cc:729] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:35:59.683513 17500 raft_consensus.cc:2726] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:35:59.696781 17255 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:35:59.654264 17456 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.693351 17478 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:35:59.720381 17296 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:35:59.702319 17481 raft_consensus.cc:2726] T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:59.717361 17458 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:59.766098 14595 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.703811 17455 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:59.768710 17502 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
W20260806 20:35:59.769654 17515 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260806 20:35:59.769901 17137 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.770177 15151 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:35:59.770735 17527 raft_consensus.cc:2781] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election won for term 8
W20260806 20:35:59.773113 17331 consensus_meta.cc:287] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.938s user 0.008s sys 0.004s
W20260806 20:35:59.773380 14518 connection.cc:232] RPC call timeout handler was delayed by -2.125140180985909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
I20260806 20:35:59.721675 17258 raft_consensus.cc:2726] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:35:59.721800 17241 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.724561 17276 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:35:59.683759 17464 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.661942 17465 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.721400 17302 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:35:59.774227 17478 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:35:59.810073 14633 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 20:35:59.916556 14518 connection.cc:232] RPC call timeout handler was delayed by -2.0999401809822302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260806 20:35:59.917158 14518 connection.cc:232] RPC call timeout handler was delayed by -2.0180401809921023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16490000000000002s for a response
W20260806 20:35:59.917582 14518 connection.cc:232] RPC call timeout handler was delayed by -1.9928401809884235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
W20260806 20:35:59.887698 17494 log.cc:924] Time spent T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.291s user 0.000s sys 0.001s
W20260806 20:35:59.888104 17501 log.cc:924] Time spent T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.272s user 0.000s sys 0.001s
W20260806 20:35:59.889362 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 187739us
W20260806 20:35:59.891070 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 109039us
W20260806 20:35:59.926124 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 213241us
W20260806 20:35:59.918087 14518 connection.cc:232] RPC call timeout handler was delayed by -1.9550401809974574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260806 20:35:59.832840 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 65039.1us
I20260806 20:35:59.728355 17374 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:35:59.928121 14518 connection.cc:232] RPC call timeout handler was delayed by -1.9217401809873991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260806 20:35:59.932046 14518 connection.cc:232] RPC call timeout handler was delayed by -1.7966401809826493s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
W20260806 20:35:59.932575 14518 connection.cc:232] RPC call timeout handler was delayed by -1.7912401809880976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
W20260806 20:35:59.932958 14518 connection.cc:232] RPC call timeout handler was delayed by -1.7651401809998788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
I20260806 20:35:59.780537 17344 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7.
W20260806 20:35:59.934571 14518 connection.cc:232] RPC call timeout handler was delayed by -1.7426401810080279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20260806 20:35:59.934983 14518 connection.cc:232] RPC call timeout handler was delayed by -1.7354401809861884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
W20260806 20:35:59.935333 14518 connection.cc:232] RPC call timeout handler was delayed by -1.6814401809824631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20230000000000004s for a response
I20260806 20:35:59.730129 17459 consensus_queue.cc:237] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.806452 17499 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.327s
I20260806 20:35:59.917457 17235 consensus_queue.cc:260] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:35:59.934095 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 168102us
W20260806 20:36:00.243811 14518 connection.cc:232] RPC call timeout handler was delayed by -1.5284401809913106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
W20260806 20:36:00.071429 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 149735us
I20260806 20:36:00.102300 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1583) took 2063 ms. Trace:
W20260806 20:36:00.244573 14518 connection.cc:232] RPC call timeout handler was delayed by -1.3178401810000651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24270000000000003s for a response
I20260806 20:36:00.114801 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1656) took 2649 ms. Trace:
I20260806 20:35:59.955770 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:35:59.999328 14521 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:00.052902 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:36:00.170713 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 400660us
I20260806 20:36:00.247722 14552 rpcz_store.cc:276] 0806 20:35:57.465363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.233312 (+767949us) service_pool.cc:227] Handling call
0806 20:35:58.378365 (+145053us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:59.529845 (+1151480us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:00.114506 (+584661us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400357354. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:00.114727 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2607273856}
W20260806 20:36:00.250658 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 295003us
I20260806 20:36:00.122582 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1663) took 1796 ms. Trace:
I20260806 20:35:59.871901 17185 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:59.915956 17403 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.245048 14629 rpcz_store.cc:276] 0806 20:35:58.039219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.039417 (+ 198us) service_pool.cc:227] Handling call
0806 20:35:58.121379 (+ 81962us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:58.803085 (+681706us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.803119 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:58.833815 (+ 30696us) log.cc:842] Serialized 26 byte log entry
0806 20:35:59.358533 (+524718us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:35:59.358574 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.359204 (+ 630us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.102057 (+742853us) raft_consensus.cc:1671] finished
0806 20:36:00.102080 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.102238 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":296987,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":1118819,"threads_started":2,"wal-append.queue_time_us":580812}
I20260806 20:35:59.782975 17255 raft_consensus.cc:686] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:36:00.105361 17509 log.cc:924] Time spent T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.430s user 0.000s sys 0.001s
I20260806 20:35:59.892063 17473 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:00.274294 14630 raft_consensus.cc:1226] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260806 20:36:00.108176 17263 consensus_meta.cc:287] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.742s user 0.006s sys 0.002s
I20260806 20:36:00.288107 14469 rpcz_store.cc:276] 0806 20:35:58.326032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.342493 (+ 16461us) service_pool.cc:227] Handling call
0806 20:35:58.409303 (+ 66810us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:58.490044 (+ 80741us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.490083 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:58.525914 (+ 35831us) log.cc:842] Serialized 52 byte log entry
0806 20:35:59.200021 (+674107us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:59.200095 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.200184 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.122269 (+922085us) raft_consensus.cc:1671] finished
0806 20:36:00.122301 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.122503 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120273,"raft.queue_time_us":85124,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":13262,"thread_start_us":550511,"threads_started":1,"wal-append.queue_time_us":866096}
I20260806 20:36:00.256264 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:35:59.932229 17482 raft_consensus.cc:2726] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:35:59.985483 17527 raft_consensus.cc:686] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:00.319077 14474 raft_consensus.cc:1226] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
W20260806 20:36:00.280190 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 350855us
I20260806 20:36:00.294034 14518 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:00.099888 17457 consensus_meta.cc:287] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.588s user 0.002s sys 0.005s
W20260806 20:35:59.985649 17503 log.cc:924] Time spent T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.386s user 0.000s sys 0.001s
I20260806 20:36:00.345330 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:00.294735 17393 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:35:59.923158 17478 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:00.107771 17507 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.490s user 0.000s sys 0.001s
I20260806 20:35:59.989014 17269 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:36:00.092319 17513 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.395s user 0.000s sys 0.000s
I20260806 20:36:00.128346 17147 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:00.106213 17456 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:35:59.838934 17524 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.569s
I20260806 20:36:00.279516 17111 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:36:00.405241 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 471620us
I20260806 20:36:00.101715 17455 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.343935 14476 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:00.101904 17462 leader_election.cc:288] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.358206 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:00.091605 17498 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:00.097259 17486 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.112679 17352 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:35:59.985782 17160 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:36:00.295858 17241 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:36:00.363911 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 107765us
I20260806 20:36:00.423147 14595 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:00.318049 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.424427 14552 raft_consensus.cc:2396] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:36:00.095606 17522 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:36:00.370792 14630 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:36:00.380344 17528 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.502s user 0.000s sys 0.001s
I20260806 20:36:00.374981 14479 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
W20260806 20:36:00.447145 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 153206us
I20260806 20:36:00.406358 17111 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:35:59.932104 17467 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.294245 17235 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:00.064065 17353 consensus_queue.cc:1035] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.279s
I20260806 20:36:00.294150 17185 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:35:59.807621 17475 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.503s
I20260806 20:36:00.397469 15151 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:00.398305 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.397434 17393 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.427214 17302 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.415373 17465 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.386057 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:00.343115 16844 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:00.343252 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:00.003607 17276 consensus_queue.cc:1035] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.382s
I20260806 20:36:00.343294 17376 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:00.411918 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:35:59.953931 17258 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.445s
I20260806 20:36:00.362696 17534 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:36:00.401415 17468 consensus_meta.cc:287] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.307s user 0.008s sys 0.005s
I20260806 20:35:59.928819 17331 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.533110 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1675) took 1783 ms. Trace:
I20260806 20:36:00.533514 16844 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.533771 14550 rpcz_store.cc:276] 0806 20:35:58.749560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.012192 (+262632us) service_pool.cc:227] Handling call
0806 20:35:59.014945 (+ 2753us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:59.049598 (+ 34653us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:59.049645 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:59.063291 (+ 13646us) log.cc:842] Serialized 26 byte log entry
0806 20:35:59.498533 (+435242us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:35:59.498589 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.498681 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.532872 (+1034191us) raft_consensus.cc:1671] finished
0806 20:36:00.532894 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.533045 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10487,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":8660,"thread_start_us":320283,"threads_started":1,"wal-append.queue_time_us":487554}
I20260806 20:36:00.535413 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.535712 16844 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:00.536013 14550 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:00.538015 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1589) took 2496 ms. Trace:
I20260806 20:36:00.539215 17331 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.538391 14631 rpcz_store.cc:276] 0806 20:35:58.041372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.041493 (+ 121us) service_pool.cc:227] Handling call
0806 20:35:58.041859 (+ 366us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:35:58.747818 (+705959us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.747851 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:35:58.804806 (+ 56955us) log.cc:842] Serialized 52 byte log entry
0806 20:35:59.326139 (+521333us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:59.326195 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.326289 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.537775 (+1211486us) raft_consensus.cc:1671] finished
0806 20:36:00.537801 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.537957 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":304324,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":1038282,"threads_started":2,"wal-append.queue_time_us":637444}
I20260806 20:36:00.382943 17457 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:00.539920 17376 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.540750 17457 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.380093 17473 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:00.448045 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:00.513271 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 424) took 1442 ms. Trace:
I20260806 20:36:00.524811 14518 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:00.343389 17263 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.540307 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.518953 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1590) took 1015 ms. Trace:
I20260806 20:36:00.406536 17464 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.455278 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:35:59.935993 17505 consensus_queue.cc:237] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.004380 17535 consensus_queue.cc:1035] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.439s
I20260806 20:36:00.397516 17523 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 20:36:00.558537 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 161166us
W20260806 20:36:00.463304 14359 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 7f41753baced49a1997940dfc28b2e00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260806 20:36:00.543614 14629 raft_consensus.cc:2379] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:00.471966 14470 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
W20260806 20:36:00.559191 14359 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 173945us
I20260806 20:36:00.544981 14474 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:00.544426 14626 rpcz_store.cc:276] 0806 20:35:59.503167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.503391 (+ 224us) service_pool.cc:227] Handling call
0806 20:35:59.503790 (+ 399us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:59.562876 (+ 59086us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:59.562907 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:35:59.562946 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:35:59.562979 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:00.518695 (+955716us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b540039f3d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:00.518739 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.518891 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135429376}
I20260806 20:36:00.528430 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1651) took 3854 ms. Trace:
I20260806 20:35:59.928565 17137 consensus_queue.cc:1035] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.398s
I20260806 20:36:00.470343 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1676) took 1720 ms. Trace:
I20260806 20:36:00.544432 14631 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:36:00.543043 14628 rpcz_store.cc:276] 0806 20:35:59.071173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.071334 (+ 161us) service_pool.cc:227] Handling call
0806 20:36:00.513080 (+1441746us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400161a54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:00.513208 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":626868608}
I20260806 20:36:00.458590 17269 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:00.554356 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.426544 17374 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.569388 14479 raft_consensus.cc:1226] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:00.573366 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.560061 14547 rpcz_store.cc:276] 0806 20:35:56.674072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:57.766571 (+1092499us) service_pool.cc:227] Handling call
0806 20:35:57.792900 (+ 26329us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:58.734038 (+941138us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:58.734082 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:58.762687 (+ 28605us) log.cc:842] Serialized 26 byte log entry
0806 20:35:59.336960 (+574273us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:35:59.337000 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.359023 (+ 22023us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.526973 (+1167950us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b24000ed968. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0423cf4 00007fb9b0421dfb 00007fb9b04b0559 00007fb9b04b01e5 00007fb9b04af740 00007fb9b31a2578 00007fb9b31a2cd8 00007fb9b317a6e8 00007fb9b3175aea 00007fb9b3174f36
0806 20:36:00.528088 (+ 1115us) raft_consensus.cc:1671] finished
0806 20:36:00.528111 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.528343 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":607561,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":70713728,"thread_start_us":1287778,"threads_started":2,"wal-append.queue_time_us":716586}
I20260806 20:36:00.542073 17376 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:00.575392 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:00.462332 17241 raft_consensus.cc:686] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:00.458315 17486 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:00.514261 17471 log.cc:924] Time spent T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 1.593s user 0.007s sys 0.004s
W20260806 20:36:00.514184 17296 consensus_meta.cc:287] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.675s user 0.010s sys 0.000s
I20260806 20:36:00.462862 17111 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:36:00.542074 17510 log.cc:924] Time spent T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.290s user 0.002s sys 0.000s
I20260806 20:36:00.580966 14550 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:00.522624 17312 consensus_meta.cc:287] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.153s user 0.013s sys 0.000s
W20260806 20:36:00.516604 17514 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.588s user 0.000s sys 0.001s
I20260806 20:36:00.567651 17462 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:00.524135 17537 consensus_queue.cc:1035] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260806 20:36:00.602645 17523 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:00.572850 14551 rpcz_store.cc:276] 0806 20:35:58.749881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.013146 (+263265us) service_pool.cc:227] Handling call
0806 20:35:59.042810 (+ 29664us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:59.048380 (+ 5570us) raft_consensus.cc:1520] Early marking committed up to index 1
0806 20:35:59.048412 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:59.048856 (+ 444us) log.cc:842] Serialized 26 byte log entry
0806 20:35:59.468881 (+420025us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:35:59.468925 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.469002 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.470096 (+1001094us) raft_consensus.cc:1671] finished
0806 20:36:00.470122 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.470270 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":399381,"threads_started":1,"wal-append.queue_time_us":499879}
I20260806 20:36:00.592578 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:00.596781 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 121664us
I20260806 20:36:00.591605 14630 raft_consensus.cc:2454] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:00.596714 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.603413 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.592823 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.553316 17468 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:36:00.685050 14628 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:00.597096 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:00.686784 14479 raft_consensus.cc:2354] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
W20260806 20:36:00.605882 17545 log.cc:924] Time spent T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.219s user 0.002s sys 0.000s
I20260806 20:36:00.606164 17185 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:00.607136 14475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1665) took 1798 ms (client timeout 2054 ms). Trace:
I20260806 20:36:00.607512 17475 raft_consensus.cc:3032] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:36:00.705845 17475 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:36:00.707504 17552 log.cc:924] Time spent T e391f934d63c4ad788870d7abc39a79c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260806 20:36:00.706969 17475 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.596581 17465 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:36:00.597774 17544 log.cc:924] Time spent T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20260806 20:36:00.608467 17464 raft_consensus.cc:2726] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:36:00.710798 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1679) took 1134 ms. Trace:
I20260806 20:36:00.593487 17548 consensus_queue.cc:1035] T ac95ea9d55914ec6912deffb9a91d8f8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.158s
I20260806 20:36:00.525094 17527 consensus_queue.cc:237] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.708757 17465 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.715556 17465 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.521395 16962 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.611069 17263 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.669054 17269 raft_consensus.cc:686] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:00.717676 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1680) took 1140 ms. Trace:
I20260806 20:36:00.718075 14554 rpcz_store.cc:276] 0806 20:35:59.577632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.705774 (+128142us) service_pool.cc:227] Handling call
0806 20:35:59.706147 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:59.981862 (+275715us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:00.047824 (+ 65962us) log.cc:842] Serialized 14 byte log entry
0806 20:36:00.642665 (+594841us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:00.642738 (+ 73us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:00.642796 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:00.717446 (+ 74650us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54000b31d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:00.717505 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.717615 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20519,"prepare.queue_time_us":65868,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":60315,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":17094,"spinlock_wait_cycles":732379904,"thread_start_us":593764,"threads_started":2,"wal-append.queue_time_us":557056}
I20260806 20:36:00.720016 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.612208 15151 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:00.447064 17352 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.695433 17462 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:00.721143 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 119718us
I20260806 20:36:00.721843 15151 leader_election.cc:302] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:00.722370 17352 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:00.582640 17528 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260806 20:36:00.669431 17296 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.723363 17491 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:00.724066 17491 raft_consensus.cc:2781] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:00.724319 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1777) took 1971 ms. Trace:
I20260806 20:36:00.724395 17491 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:00.724718 17296 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.724874 17491 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:00.725610 14624 raft_consensus.cc:2379] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:00.725363 17546 consensus_queue.cc:1035] T 500956bd1a3846a3a7b16c51ddecbc44 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:36:00.726048 17475 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:00.596926 17469 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:00.593560 17302 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:00.743184 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 46845us
I20260806 20:36:00.724758 14553 rpcz_store.cc:276] 0806 20:35:58.752866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.032831 (+279965us) service_pool.cc:227] Handling call
0806 20:35:59.033280 (+ 449us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:59.077985 (+ 44705us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:35:59.497352 (+419367us) log.cc:842] Serialized 14 byte log entry
0806 20:35:59.669199 (+171847us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:59.687586 (+ 18387us) log.cc:842] Serialized 26 byte log entry
0806 20:35:59.687753 (+ 167us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:35:59.687784 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:35:59.687851 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.724118 (+1036267us) raft_consensus.cc:1671] finished
0806 20:36:00.724145 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.724267 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17397,"prepare.queue_time_us":413945,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":44003,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":24857,"thread_start_us":437938,"threads_started":2,"wal-append.queue_time_us":203121}
I20260806 20:36:00.680886 14547 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:00.689728 14627 raft_consensus.cc:2379] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:00.689873 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:00.419575 17498 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:00.751803 15151 connection.cc:232] RPC call timeout handler was delayed by -1.1958238149818499s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260806 20:36:00.702306 17549 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.108s user 0.000s sys 0.002s
I20260806 20:36:00.560333 17534 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.694654 17457 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.684701 17517 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:00.705242 14596 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:00.708709 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1597) took 1160 ms. Trace:
I20260806 20:36:00.711236 14555 rpcz_store.cc:276] 0806 20:35:59.577336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.697401 (+120065us) service_pool.cc:227] Handling call
0806 20:35:59.721977 (+ 24576us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:59.722208 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:59.722241 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:59.728079 (+ 5838us) log.cc:842] Serialized 26 byte log entry
0806 20:36:00.004195 (+276116us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:00.004242 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:00.004312 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.710524 (+706212us) raft_consensus.cc:1671] finished
0806 20:36:00.710554 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.710725 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36736,"thread_start_us":230439,"threads_started":1,"wal-append.queue_time_us":595738}
I20260806 20:36:00.609527 17393 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:00.722528 17462 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:00.580229 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:36:00.753890 17466 consensus_meta.cc:287] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.264s user 0.012s sys 0.000s
I20260806 20:36:00.770179 14479 raft_consensus.cc:1261] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:36:00.728022 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:00.729508 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:00.779794 15187 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:00.749475 14550 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:00.696877 14548 raft_consensus.cc:2396] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 20:36:00.469043 17255 consensus_queue.cc:237] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.753400 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.759441 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.757155 14625 rpcz_store.cc:276] 0806 20:35:59.548648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.548867 (+ 219us) service_pool.cc:227] Handling call
0806 20:35:59.572494 (+ 23627us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:59.921704 (+349210us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:59.921756 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:59.922308 (+ 552us) log.cc:842] Serialized 26 byte log entry
0806 20:36:00.398572 (+476264us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:00.398613 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:00.514868 (+116255us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400180bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:00.514915 (+ 47us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.706869 (+191954us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:00.708521 (+ 1652us) raft_consensus.cc:1671] finished
0806 20:36:00.708544 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.708657 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113033,"raft.queue_time_us":124877,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":29891,"spinlock_wait_cycles":633420928,"thread_start_us":412433,"threads_started":1,"wal-append.queue_time_us":677124}
W20260806 20:36:00.760260 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 55134.3us
I20260806 20:36:00.784226 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:00.784910 14470 raft_consensus.cc:3037] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:00.755607 17480 raft_consensus.cc:482] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:00.763062 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:00.776964 17386 consensus_meta.cc:287] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.046s user 0.012s sys 0.000s
W20260806 20:36:00.694712 14475 rpcz_store.cc:269] 0806 20:35:58.808318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.808527 (+ 209us) service_pool.cc:227] Handling call
0806 20:36:00.606904 (+1798377us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b5400332cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:00.607056 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1331165440}
I20260806 20:36:00.762918 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:00.766824 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.743065 17469 raft_consensus.cc:482] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:00.767403 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:00.767333 17543 consensus_queue.cc:1035] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260806 20:36:00.768986 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1678) took 1193 ms. Trace:
I20260806 20:36:00.800573 14555 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:36:00.805794 14549 rpcz_store.cc:276] 0806 20:35:59.575670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.673499 (+ 97829us) service_pool.cc:227] Handling call
0806 20:35:59.694190 (+ 20691us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:35:59.925666 (+231476us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:35:59.925701 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:35:59.952497 (+ 26796us) log.cc:842] Serialized 26 byte log entry
0806 20:36:00.390646 (+438149us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:00.390711 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:00.512597 (+121886us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54003e80d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:00.512648 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:00.611557 (+ 98909us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:00.768781 (+157224us) raft_consensus.cc:1671] finished
0806 20:36:00.768805 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:00.768931 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":570569984,"thread_start_us":317022,"threads_started":1,"wal-append.queue_time_us":556226}
I20260806 20:36:00.692003 17523 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.757297 17137 consensus_queue.cc:1035] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260806 20:36:00.779872 14551 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:00.478809 17522 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:00.781325 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.786571 14628 raft_consensus.cc:2379] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:00.787367 14596 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:00.662173 17486 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.766248 14626 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:00.756134 16962 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:00.803149 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:00.718075 17269 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.753401 17302 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.813757 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:00.808876 17543 consensus_queue.cc:1225] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer 148bf2a11dba423ab364371f0458e871 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 20:36:00.818945 17559 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:00.783433 17393 consensus_queue.cc:1046] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260806 20:36:00.797920 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:00.820569 14552 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:00.782498 17466 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:00.802335 17386 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.789366 17241 consensus_queue.cc:237] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:00.799010 17517 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:00.901161 14444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.002s sys 0.039s
W20260806 20:36:01.124084 14444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.741s user 0.007s sys 0.064s
I20260806 20:36:01.101737 15150 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:36:00.990855 14633 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.030s sys 0.028s
W20260806 20:36:01.208330 14633 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s user 0.040s sys 0.040s
I20260806 20:36:00.903858 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:01.110440 14551 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:00.821643 14596 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:01.210039 14631 raft_consensus.cc:2354] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:36:00.901463 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:01.125497 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 341597us
I20260806 20:36:01.211172 14549 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 3.
W20260806 20:36:00.851294 14441 leader_election.cc:334] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.595s (SENT)
I20260806 20:36:01.207916 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:01.161825 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 684383us
I20260806 20:36:01.213953 14550 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:36:01.214084 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:01.214202 14518 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:01.211443 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 389879us
I20260806 20:36:01.213222 14441 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:01.187245 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1677) took 1618 ms. Trace:
I20260806 20:36:00.918542 14470 raft_consensus.cc:2454] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:36:01.215592 14548 rpcz_store.cc:276] 0806 20:35:59.569295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.665230 (+ 95935us) service_pool.cc:227] Handling call
0806 20:36:01.187024 (+1521794us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540018f1d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:01.187182 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2360619904}
I20260806 20:36:00.803469 17469 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:00.870564 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:01.211036 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:01.219643 14596 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:01.266901 14624 raft_consensus.cc:1261] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:36:01.209663 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1780) took 1222 ms. Trace:
I20260806 20:36:01.267951 14551 rpcz_store.cc:276] 0806 20:35:59.987634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.681666 (+694032us) service_pool.cc:227] Handling call
0806 20:36:01.209584 (+527918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:01.268888 14553 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:01.221166 14521 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:01.270578 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 56433.4us
I20260806 20:36:00.894869 14554 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
W20260806 20:36:01.102876 17562 log.cc:924] Time spent T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 20:36:00.950377 14474 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:36:01.209683 14555 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
W20260806 20:36:00.940698 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 45327.8us
I20260806 20:36:01.176157 14547 raft_consensus.cc:2379] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:36:00.958985 15181 raft_consensus.cc:2379] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:36:01.284384 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:01.048240 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:36:01.001420 17478 consensus_meta.cc:287] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.585s user 0.009s sys 0.000s
I20260806 20:36:01.285943 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:01.286453 14629 raft_consensus.cc:1226] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:01.287035 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:01.292649 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:01.293475 14549 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:36:01.293893 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:01.294476 14551 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 6
I20260806 20:36:01.294340 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:01.103066 17529 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:36:00.999301 17550 log.cc:924] Time spent T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
I20260806 20:36:01.305542 17529 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:01.306078 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:01.307210 14555 raft_consensus.cc:2396] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 8 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 20:36:01.004647 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:01.308976 15178 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:36:01.309046 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1601) took 1036 ms. Trace:
W20260806 20:36:00.902762 17520 log.cc:924] Time spent T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 1.224s user 0.009s sys 0.003s
I20260806 20:36:01.310045 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:01.299566 17498 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:01.185771 17560 log.cc:924] Time spent T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260806 20:36:01.311478 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:01.306079 17529 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:00.860597 17522 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:01.313197 17517 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:01.314481 17522 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:01.315053 17302 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:00.896471 17500 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:01.316133 17500 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:01.316569 17500 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:00.985267 16844 consensus_queue.cc:1046] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.334s
I20260806 20:36:01.318485 16844 raft_consensus.cc:3032] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:01.318828 16844 raft_consensus.cc:729] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:01.319639 16844 consensus_queue.cc:260] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:01.185066 17554 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:36:01.320781 17554 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:01.321159 17554 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:01.325752 17386 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:01.165666 17557 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:01.326615 17557 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:01.327641 14553 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
W20260806 20:36:01.287688 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 66632.9us
I20260806 20:36:01.329471 17466 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:01.329231 17529 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:01.242136 16962 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:01.126138 17559 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:01.101823 17546 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:01.300582 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 449328us
I20260806 20:36:01.291299 14624 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:36:01.304759 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:01.100837 17530 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:01.308199 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:00.998755 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:01.335860 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:01.312299 15178 raft_consensus.cc:2354] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6.
I20260806 20:36:01.165894 17486 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:01.354423 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:00.813011 15180 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:36:01.102748 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:00.820155 17534 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:01.267318 17527 raft_consensus.cc:2781] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:01.348310 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:01.285385 17504 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:01.434417 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:01.309616 14633 rpcz_store.cc:276] 0806 20:36:00.272631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.272800 (+ 169us) service_pool.cc:227] Handling call
0806 20:36:00.273239 (+ 439us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:00.319108 (+ 45869us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:00.319145 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:00.335257 (+ 16112us) log.cc:842] Serialized 26 byte log entry
0806 20:36:01.209263 (+874006us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:01.209302 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:01.209379 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:01.308780 (+ 99401us) raft_consensus.cc:1671] finished
0806 20:36:01.308808 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:01.308973 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46522,"raft.queue_time_us":61128,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":9232,"thread_start_us":805968,"threads_started":1,"wal-append.queue_time_us":952515}
W20260806 20:36:01.297214 14630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.009s sys 0.008s
I20260806 20:36:00.807332 17480 raft_consensus.cc:504] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } attrs { replace: true } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:01.284724 17465 raft_consensus.cc:2726] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:36:01.441174 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1667) took 3864 ms. Trace:
I20260806 20:36:01.456383 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:00.811430 17457 consensus_queue.cc:1046] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.536s
I20260806 20:36:01.433543 17529 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:02.036345 14549 consensus_meta.cc:287] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.697s user 0.007s sys 0.002s
W20260806 20:36:01.762694 15151 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.711s user 0.006s sys 0.057s
W20260806 20:36:02.039283 15151 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.988s user 0.017s sys 0.063s
I20260806 20:36:02.039232 14549 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
W20260806 20:36:02.039691 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 988726us
I20260806 20:36:02.039871 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1698) took 1088 ms. Trace:
I20260806 20:36:02.040246 14549 rpcz_store.cc:276] 0806 20:36:00.951491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.212926 (+261435us) service_pool.cc:227] Handling call
0806 20:36:02.039815 (+826889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.040905 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:02.041756 14555 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:36:02.040249 15151 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 2.190s (SENT) [suppressed 11 similar messages]
I20260806 20:36:02.041983 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:02.043462 14553 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:02.042632 14546 rpcz_store.cc:276] 0806 20:35:57.576313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:58.736036 (+1159723us) service_pool.cc:227] Handling call
0806 20:35:58.761622 (+ 25586us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:35:59.055364 (+293742us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:36:00.524137 (+1468773us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b180047e9c0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b318282c 00007fb9b05ad9dc 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05c8155 00007fb9b6ec8167 00007fb9b34fd807 00007fb9b350cf0f
0806 20:36:00.524279 (+ 142us) log.cc:842] Serialized 14 byte log entry
0806 20:36:01.046243 (+521964us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:01.046480 (+ 237us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:01.046669 (+ 189us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:01.440748 (+394079us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54002db4d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:01.440822 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:01.441101 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67456,"prepare.queue_time_us":402937,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":70534,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":17839,"spinlock_wait_cycles":705604736,"thread_start_us":841193,"threads_started":2,"wal-append.queue_time_us":661490}
I20260806 20:36:02.044245 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1803) took 1338 ms. Trace:
I20260806 20:36:02.045361 14553 rpcz_store.cc:276] 0806 20:36:00.706076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.354184 (+648108us) service_pool.cc:227] Handling call
0806 20:36:02.044177 (+689993us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1531136}
W20260806 20:36:01.975819 14628 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.000s sys 0.010s
I20260806 20:36:01.439198 14629 raft_consensus.cc:2354] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
W20260806 20:36:02.046509 14628 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s user 0.000s sys 0.011s
I20260806 20:36:01.442497 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:01.505695 14441 leader_election.cc:334] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.538s (SENT)
W20260806 20:36:02.049857 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 544213us
I20260806 20:36:02.051024 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:01.809331 14551 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
W20260806 20:36:02.052211 14630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.266s user 0.009s sys 0.034s
I20260806 20:36:02.052206 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 408) took 1100 ms. Trace:
W20260806 20:36:01.457360 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 101721us
I20260806 20:36:02.052938 14551 rpcz_store.cc:276] 0806 20:36:00.951765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.293642 (+341877us) service_pool.cc:227] Handling call
0806 20:36:02.052137 (+758495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.054358 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:02.045509 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 712745us
W20260806 20:36:01.944434 14596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.003s sys 0.017s
W20260806 20:36:02.060298 14596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.788s user 0.008s sys 0.017s
W20260806 20:36:02.060748 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 841162us
I20260806 20:36:02.060936 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.047325 14550 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:36:02.062556 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:02.062832 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1700) took 1006 ms. Trace:
I20260806 20:36:02.063302 14549 raft_consensus.cc:2379] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:36:02.063753 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1789) took 1727 ms. Trace:
I20260806 20:36:02.063138 14550 rpcz_store.cc:276] 0806 20:36:01.056914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.302653 (+245739us) service_pool.cc:227] Handling call
0806 20:36:02.062780 (+760127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.064219 14549 rpcz_store.cc:276] 0806 20:36:00.335957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.062311 (+1726354us) service_pool.cc:227] Handling call
0806 20:36:02.063702 (+ 1391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.063706 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:02.065033 14553 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:36:02.065076 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:02.065701 14549 raft_consensus.cc:2454] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:02.066267 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1699) took 1081 ms. Trace:
I20260806 20:36:02.062094 14596 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:01.294843 17537 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.066629 14549 rpcz_store.cc:276] 0806 20:36:00.984678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.064832 (+1080154us) service_pool.cc:227] Handling call
0806 20:36:02.066207 (+ 1375us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:01.299698 17393 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:36:01.295578 17473 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260806 20:36:01.951121 17455 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.006s sys 0.014s
I20260806 20:36:02.068324 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 653) took 1248 ms. Trace:
I20260806 20:36:01.479760 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:02.085989 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:01.295187 17235 raft_consensus.cc:2726] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:01.299636 17462 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:02.072045 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 401) took 1309 ms. Trace:
I20260806 20:36:02.103108 14555 rpcz_store.cc:276] 0806 20:36:00.762474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.434392 (+671918us) service_pool.cc:227] Handling call
0806 20:36:02.071959 (+637567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.072803 14624 raft_consensus.cc:2354] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6.
W20260806 20:36:02.093035 15151 connection.cc:232] RPC call timeout handler was delayed by -1.0382860389945563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32400000000000007s for a response
I20260806 20:36:02.075054 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:02.073613 14444 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:02.079728 14631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1886) took 1655 ms (client timeout 2117 ms). Trace:
I20260806 20:36:02.108680 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 425) took 1314 ms. Trace:
I20260806 20:36:02.101127 17543 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.077410 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.110574 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:02.110473 17543 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.111150 14546 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:36:02.111387 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.112947 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:02.113564 14549 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:02.080962 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 402) took 1300 ms. Trace:
I20260806 20:36:02.115075 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1682) took 2536 ms. Trace:
I20260806 20:36:02.114945 14551 rpcz_store.cc:276] 0806 20:36:00.781858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.054054 (+1272196us) service_pool.cc:227] Handling call
0806 20:36:02.080885 (+ 26831us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.115526 14549 rpcz_store.cc:276] 0806 20:35:59.578209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.114234 (+2536025us) service_pool.cc:227] Handling call
0806 20:36:02.114525 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.114691 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.114729 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:02.114787 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:02.114837 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.114912 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.115026 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.116745 14551 pending_rounds.cc:76] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 2
I20260806 20:36:01.331985 17478 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.117125 14551 pending_rounds.cc:97] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 20:36:02.117695 14551 raft_consensus.cc:2866] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:36:02.117888 14632 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:02.118000 14551 pending_rounds.cc:97] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260806 20:36:02.118273 14551 raft_consensus.cc:2866] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:36:02.118949 17478 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:01.979552 17563 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
W20260806 20:36:01.966246 17554 consensus_meta.cc:287] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.645s user 0.006s sys 0.005s
I20260806 20:36:01.518210 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:02.120004 17554 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.121094 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 658) took 1136 ms. Trace:
I20260806 20:36:02.121593 15180 rpcz_store.cc:276] 0806 20:36:00.984261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.517860 (+533599us) service_pool.cc:227] Handling call
0806 20:36:02.121010 (+603150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.122514 17554 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:02.123106 17554 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:02.123008 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:36:02.032572 17572 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260806 20:36:02.086762 15185 rpcz_store.cc:276] 0806 20:36:00.820348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.456092 (+635744us) service_pool.cc:227] Handling call
0806 20:36:02.068246 (+612154us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:01.535096 17491 consensus_meta.cc:287] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.810s user 0.003s sys 0.007s
I20260806 20:36:02.078409 14548 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:01.433898 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:36:02.088704 14554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1781) took 2100 ms (client timeout 2225 ms). Trace:
W20260806 20:36:02.133250 14554 rpcz_store.cc:269] 0806 20:35:59.988008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.719762 (+731754us) service_pool.cc:227] Handling call
0806 20:36:02.088479 (+1368717us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b5400086bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:02.088625 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":371576192}
I20260806 20:36:02.133881 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:02.134666 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:01.922536 17535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.039s sys 0.075s
W20260806 20:36:02.136519 17535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s user 0.043s sys 0.077s
I20260806 20:36:02.100567 14518 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:02.137781 14518 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:02.138017 17505 raft_consensus.cc:3032] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Stepping down as leader of term 7
W20260806 20:36:01.536441 17111 consensus_meta.cc:287] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.851s user 0.009s sys 0.000s
I20260806 20:36:02.138370 17505 raft_consensus.cc:729] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:02.138691 17111 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.139169 17505 consensus_queue.cc:260] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.140250 17491 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.141386 17111 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.101516 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:02.142118 14550 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:36:02.142369 17491 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:02.135394 14554 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:36:02.143224 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:02.143833 15185 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:36:02.146370 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1789) took 1496 ms. Trace:
I20260806 20:36:02.146875 14474 rpcz_store.cc:276] 0806 20:36:00.649652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.674256 (+ 24604us) service_pool.cc:227] Handling call
0806 20:36:02.146153 (+1471897us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b5400328054. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:02.146294 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":472940800}
I20260806 20:36:02.149008 15182 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20260806 20:36:01.932164 17475 consensus_meta.cc:287] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.206s user 0.010s sys 0.003s
I20260806 20:36:01.522279 17523 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.082513 14596 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:02.109323 15187 rpcz_store.cc:276] 0806 20:36:00.794752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.794920 (+ 168us) service_pool.cc:227] Handling call
0806 20:36:02.074811 (+1279891us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b5800005498. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:36:02.108600 (+ 33789us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6373120}
I20260806 20:36:02.109993 17465 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:02.099154 17455 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.793s user 0.006s sys 0.020s
I20260806 20:36:01.509837 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:02.069033 14547 raft_consensus.cc:2379] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:02.086863 14627 raft_consensus.cc:2379] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
W20260806 20:36:02.108757 14631 rpcz_store.cc:269] 0806 20:36:00.424166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.573130 (+148964us) service_pool.cc:227] Handling call
0806 20:36:02.063186 (+1490056us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b5400312754. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:02.079650 (+ 16464us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1457822976}
I20260806 20:36:02.037236 17486 consensus_queue.cc:1046] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260806 20:36:01.433518 17530 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:02.075623 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.876s user 0.000s sys 0.013s
I20260806 20:36:02.128222 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.061858 17185 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.070636 17302 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:01.532351 17469 raft_consensus.cc:504] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.183863 15185 raft_consensus.cc:2396] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:36:02.162487 14629 raft_consensus.cc:2454] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:36:02.161794 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1685) took 1603 ms. Trace:
I20260806 20:36:02.187834 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 655) took 1293 ms. Trace:
I20260806 20:36:02.131354 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:01.516774 16962 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:01.685546 17557 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.067982 17403 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.152114 15181 raft_consensus.cc:2362] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6. Re-sending same reply.
I20260806 20:36:02.194432 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:36:02.194656 15181 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 652) took 1374 ms (client timeout 1538 ms). Trace:
W20260806 20:36:02.195050 15181 rpcz_store.cc:269] 0806 20:36:00.820078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.305132 (+485054us) service_pool.cc:227] Handling call
0806 20:36:02.194592 (+889460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.195730 17469 leader_election.cc:288] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:01.768649 17559 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.067384 17459 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:02.156255 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 426) took 1361 ms. Trace:
I20260806 20:36:02.196689 17459 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.197137 15179 rpcz_store.cc:276] 0806 20:36:00.795107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.797397 (+ 2290us) service_pool.cc:227] Handling call
0806 20:36:02.153963 (+1356566us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.198865 17459 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.079466 17296 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:02.199111 15179 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:02.199541 17296 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.200095 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 657) took 1240 ms. Trace:
W20260806 20:36:02.161337 17531 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.553s user 0.015s sys 0.012s
I20260806 20:36:02.194442 17403 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:02.201321 17531 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.593s user 0.016s sys 0.012s
W20260806 20:36:02.152904 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 79405.3us
I20260806 20:36:02.087715 17534 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:02.154815 14555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:36:02.202346 14555 raft_consensus.cc:595] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:02.152372 14624 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:02.156533 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:02.202708 14555 raft_consensus.cc:603] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:36:02.203236 15187 raft_consensus.cc:2379] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:36:01.526726 17500 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:01.605597 17562 log.cc:924] Time spent T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.271s user 0.000s sys 0.000s
I20260806 20:36:02.102896 17462 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:02.124120 15180 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:02.123831 17554 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:02.181221 15186 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s user 0.005s sys 0.015s
W20260806 20:36:02.213984 15186 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.971s user 0.005s sys 0.015s
W20260806 20:36:02.159349 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 76935.5us
I20260806 20:36:02.188525 14546 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:36:02.188730 14547 rpcz_store.cc:276] 0806 20:36:00.558379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.287740 (+729361us) service_pool.cc:227] Handling call
0806 20:36:02.161732 (+873992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.131225 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.217326 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1686) took 1658 ms. Trace:
I20260806 20:36:02.217949 14633 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 8.
I20260806 20:36:02.189895 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1795) took 1656 ms. Trace:
I20260806 20:36:02.218916 14553 rpcz_store.cc:276] 0806 20:36:00.533739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.086949 (+1553210us) service_pool.cc:227] Handling call
0806 20:36:02.189836 (+102887us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.219383 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 506) took 1489 ms. Trace:
I20260806 20:36:02.100129 17537 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:02.162101 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.994s user 0.000s sys 0.013s
W20260806 20:36:02.233017 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 1.1313e+06us
I20260806 20:36:02.163146 17523 raft_consensus.cc:2726] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:36:02.190485 15185 rpcz_store.cc:276] 0806 20:36:00.894860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.127762 (+1232902us) service_pool.cc:227] Handling call
0806 20:36:02.187777 (+ 60015us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.164101 17111 raft_consensus.cc:2781] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:02.087039 17566 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.203182 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48150 (request call id 28) took 1660 ms. Trace:
I20260806 20:36:02.085284 17393 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:02.203747 14624 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:02.235325 14555 rpcz_store.cc:276] 0806 20:36:00.543127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.103934 (+1560807us) service_pool.cc:227] Handling call
0806 20:36:02.203129 (+ 99195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.237269 14624 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:36:02.099766 17527 raft_consensus.cc:686] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:01.437865 17546 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.207777 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 649) took 1388 ms. Trace:
I20260806 20:36:02.250859 15187 rpcz_store.cc:276] 0806 20:36:00.819237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.156221 (+1336984us) service_pool.cc:227] Handling call
0806 20:36:02.207717 (+ 51496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.211210 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 442) took 1664 ms. Trace:
I20260806 20:36:02.252254 14442 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:02.252516 14630 rpcz_store.cc:276] 0806 20:36:00.546708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.698471 (+151763us) service_pool.cc:227] Handling call
0806 20:36:00.698929 (+ 458us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:00.758156 (+ 59227us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:00.758203 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:00.758747 (+ 544us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.052894 (+1294147us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:02.053002 (+ 108us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.053066 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:02.210996 (+157930us) raft_consensus.cc:1671] finished
0806 20:36:02.211016 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.211147 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7256,"raft.queue_time_us":14015,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":14443,"thread_start_us":1266725,"threads_started":1,"wal-append.queue_time_us":1060046}
I20260806 20:36:02.164990 17473 raft_consensus.cc:2781] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:02.263924 14442 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:02.219748 15186 rpcz_store.cc:276] 0806 20:36:00.730174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.750736 (+ 20562us) service_pool.cc:227] Handling call
0806 20:36:00.763867 (+ 13131us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:00.777570 (+ 13703us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:36:01.242546 (+464976us) log.cc:842] Serialized 14 byte log entry
0806 20:36:02.219064 (+976518us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:02.219123 (+ 59us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:02.219171 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.219233 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.219327 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4452,"prepare.queue_time_us":910749,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"thread_start_us":1400207,"threads_started":2}
I20260806 20:36:02.217787 14546 rpcz_store.cc:276] 0806 20:36:00.558727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.080540 (+1521813us) service_pool.cc:227] Handling call
0806 20:36:02.217256 (+136716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.266327 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1797) took 1731 ms. Trace:
I20260806 20:36:02.266758 14546 rpcz_store.cc:276] 0806 20:36:00.534416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.265352 (+1730936us) service_pool.cc:227] Handling call
0806 20:36:02.265743 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.265950 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.265984 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:02.266033 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:02.266077 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.266139 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.266261 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.233160 14625 raft_consensus.cc:1226] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:02.081595 17504 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:02.155964 17255 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.055s sys 0.046s
W20260806 20:36:02.218945 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.947s user 0.017s sys 0.037s
I20260806 20:36:02.249202 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:02.255831 14444 leader_election.cc:398] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6.
I20260806 20:36:02.208122 16962 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:36:02.257177 15185 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 20:36:02.203588 17543 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:02.207618 17534 raft_consensus.cc:686] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:36:02.222522 17269 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.006s sys 0.019s
I20260806 20:36:02.212329 17462 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:02.248442 17393 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:02.308339 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:02.309262 14547 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:02.225138 17332 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.004s sys 0.024s
I20260806 20:36:02.211941 17486 raft_consensus.cc:3032] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:36:02.163451 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1787) took 1883 ms. Trace:
I20260806 20:36:02.316594 14549 rpcz_store.cc:276] 0806 20:36:00.280204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.151671 (+1871467us) service_pool.cc:227] Handling call
0806 20:36:02.163029 (+ 11358us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.163178 (+ 149us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.163200 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:02.163237 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:02.163264 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.163312 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.163401 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.269770 14444 leader_election.cc:401] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260806 20:36:02.270174 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.998s user 0.017s sys 0.037s
W20260806 20:36:02.318843 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 1.10476e+06us
W20260806 20:36:02.290778 17269 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.961s user 0.007s sys 0.019s
I20260806 20:36:02.287529 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:02.325876 14546 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:36:02.327034 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 71414.5us
I20260806 20:36:02.274423 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 659) took 1268 ms. Trace:
I20260806 20:36:02.329195 15185 rpcz_store.cc:276] 0806 20:36:01.006102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.234037 (+1227935us) service_pool.cc:227] Handling call
0806 20:36:02.234411 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.274104 (+ 39693us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.274361 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.156816 17465 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.291064 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:02.222558 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 656) took 1284 ms. Trace:
I20260806 20:36:02.293217 15183 raft_consensus.cc:2396] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 20:36:02.200599 15179 rpcz_store.cc:276] 0806 20:36:00.959316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.198459 (+1239143us) service_pool.cc:227] Handling call
0806 20:36:02.198845 (+ 386us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.199768 (+ 923us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.200010 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.295042 14442 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:02.190167 17185 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.204790 17302 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:02.268477 17255 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s user 0.057s sys 0.046s
I20260806 20:36:02.260391 17455 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:02.262512 17111 raft_consensus.cc:686] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 8 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:02.321564 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:02.219558 15178 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 654) took 1393 ms. Trace:
I20260806 20:36:02.235181 17459 consensus_queue.cc:1035] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260806 20:36:02.256985 17566 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.280055 17523 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:36:02.335541 14552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1779) took 2348 ms (client timeout 1657 ms). Trace:
I20260806 20:36:02.374645 14547 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
W20260806 20:36:02.376766 14552 rpcz_store.cc:269] 0806 20:35:59.987244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.456905 (+469661us) service_pool.cc:227] Handling call
0806 20:36:02.335305 (+1878400us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b540018ddd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:02.335447 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":568578432}
I20260806 20:36:02.377800 14547 raft_consensus.cc:1226] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:02.312616 17543 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:36:02.345110 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:02.379848 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.318730 17580 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:02.346309 15184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.051s user 0.017s sys 0.056s
I20260806 20:36:02.320215 17529 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:02.322713 17486 raft_consensus.cc:729] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:02.387921 14478 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:36:02.357941 15185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:36:02.364110 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.366423 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.389174 15185 raft_consensus.cc:595] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:02.389328 14595 connection.cc:232] RPC call timeout handler was delayed by -2.9869300500140525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
I20260806 20:36:02.389599 15185 raft_consensus.cc:603] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:36:02.389730 14595 connection.cc:232] RPC call timeout handler was delayed by -1.7066252829972655s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
I20260806 20:36:02.368005 14470 raft_consensus.cc:1261] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:36:02.372337 14476 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:36:02.336133 15180 rpcz_store.cc:276] 0806 20:36:00.938315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.122625 (+1184310us) service_pool.cc:227] Handling call
0806 20:36:02.218324 (+ 95699us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.373317 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.376349 14555 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:36:02.373337 17302 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.382805 14549 raft_consensus.cc:2379] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:36:02.383363 15184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.281s user 0.017s sys 0.056s
I20260806 20:36:02.383656 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.384498 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1799) took 1849 ms. Trace:
I20260806 20:36:02.385789 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.398365 14547 rpcz_store.cc:276] 0806 20:36:00.535011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.377333 (+1842322us) service_pool.cc:227] Handling call
0806 20:36:02.377634 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.384280 (+ 6646us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.384410 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.389755 14471 raft_consensus.cc:2379] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:36:02.373369 14475 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
W20260806 20:36:02.333185 17332 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.781s user 0.004s sys 0.024s
I20260806 20:36:02.393843 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1798) took 1859 ms. Trace:
I20260806 20:36:02.401889 14555 rpcz_store.cc:276] 0806 20:36:00.534734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.345259 (+1810525us) service_pool.cc:227] Handling call
0806 20:36:02.371624 (+ 26365us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.393614 (+ 21990us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.393800 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.363373 14546 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:02.339166 17577 consensus_queue.cc:1035] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260806 20:36:02.376163 17269 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:02.404635 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.382484 17543 raft_consensus.cc:482] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:02.405738 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.408927 14476 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:02.409392 14476 raft_consensus.cc:2354] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:36:02.411005 14555 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 20:36:02.411536 14549 pending_rounds.cc:76] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 2
I20260806 20:36:02.411821 14549 pending_rounds.cc:97] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 20:36:02.412016 14547 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:02.356804 15178 rpcz_store.cc:276] 0806 20:36:00.820639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.509419 (+688780us) service_pool.cc:227] Handling call
0806 20:36:02.214415 (+704996us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.410166 17543 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:02.414878 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1689) took 1824 ms. Trace:
I20260806 20:36:02.420996 14555 rpcz_store.cc:276] 0806 20:36:00.590960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.403021 (+1812061us) service_pool.cc:227] Handling call
0806 20:36:02.410677 (+ 7656us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.411400 (+ 723us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.414817 (+ 3417us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.412070 14549 raft_consensus.cc:2866] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:36:02.415616 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1688) took 1836 ms. Trace:
I20260806 20:36:02.401054 17478 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:36:02.422269 14547 rpcz_store.cc:276] 0806 20:36:00.579547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.399147 (+1819600us) service_pool.cc:227] Handling call
0806 20:36:02.411866 (+ 12719us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.412570 (+ 704us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.415558 (+ 2988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:02.405400 17296 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:02.426620 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.447355 14521 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:02.409785 17530 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:02.409649 17403 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:02.431655 17455 raft_consensus.cc:686] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:02.490823 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1802) took 1785 ms. Trace:
I20260806 20:36:02.392889 17523 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.411257 17480 leader_election.cc:288] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.478950 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.415205 17500 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:36:02.479457 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 341746us
I20260806 20:36:02.391009 17580 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.392406 17529 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.481411 15151 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:02.433812 17255 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.356758 17527 consensus_queue.cc:237] T 6a9d79d20363456d80521feddfec49bc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.640888 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:02.496165 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.520677 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:02.427382 17478 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:02.534586 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:02.426204 17486 consensus_queue.cc:260] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.552749 15185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "92828dd6dbe74f29a04d4e702b502621"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:36:02.651474 15185 raft_consensus.cc:595] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Received request to transfer leadership to TS 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:02.651839 15185 raft_consensus.cc:603] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:36:02.475075 17554 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:02.413054 17269 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.487155 17530 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.501405 17403 raft_consensus.cc:2781] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:36:02.587615 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 353285us
W20260806 20:36:02.590744 14441 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.000s sys 0.014s
I20260806 20:36:02.478718 17557 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:02.608536 17185 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.666342 15150 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:02.502058 14521 leader_election.cc:398] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
I20260806 20:36:02.608772 14473 raft_consensus.cc:1261] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 20:36:02.667138 14521 leader_election.cc:401] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:36:02.434885 17559 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6.
I20260806 20:36:02.487322 17302 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.478608 17469 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:02.609104 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1610) took 1339 ms. Trace:
I20260806 20:36:02.388728 17534 consensus_queue.cc:237] T fefa7fe42a5543c3970808c8a8e8e45c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.613271 17523 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:02.491514 14547 rpcz_store.cc:276] 0806 20:36:00.705806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.425646 (+1719840us) service_pool.cc:227] Handling call
0806 20:36:02.490120 (+ 64474us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.490379 (+ 259us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.490415 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:02.490468 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:02.490555 (+ 87us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.490626 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.490753 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:02.617535 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 267964us
I20260806 20:36:02.519143 17537 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:02.629143 14518 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:02.631572 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 150260us
I20260806 20:36:02.546968 17504 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:36:02.771904 17575 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.399s user 0.004s sys 0.001s
I20260806 20:36:02.543339 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.644935 14471 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:02.621989 17591 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:36:02.699719 17589 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
W20260806 20:36:02.668799 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 192199us
I20260806 20:36:02.671190 14473 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:36:02.492612 17111 consensus_queue.cc:237] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.644886 17473 raft_consensus.cc:686] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:02.951205 17566 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:02.733362 14548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1702) took 1651 ms (client timeout 1714 ms). Trace:
W20260806 20:36:03.055016 14548 rpcz_store.cc:269] 0806 20:36:01.082673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.219328 (+136655us) service_pool.cc:227] Handling call
0806 20:36:02.733136 (+1513808us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b54001faed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:02.733290 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1797804416}
W20260806 20:36:03.022958 17588 log.cc:924] Time spent T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.379s user 0.000s sys 0.000s
I20260806 20:36:02.643453 17480 raft_consensus.cc:2741] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 20:36:02.935210 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.934180 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:02.952276 15185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1"
dest_uuid: "148bf2a11dba423ab364371f0458e871"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:56926
I20260806 20:36:02.672794 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:03.063912 15185 raft_consensus.cc:595] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:02.893334 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 391149us
I20260806 20:36:03.064256 15185 raft_consensus.cc:603] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:36:02.766790 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 424) took 2006 ms. Trace:
I20260806 20:36:02.765959 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 444) took 1953 ms. Trace:
I20260806 20:36:03.065528 14625 rpcz_store.cc:276] 0806 20:36:00.812115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.812363 (+ 248us) service_pool.cc:227] Handling call
0806 20:36:00.902346 (+ 89983us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.287008 (+1384662us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.765653 (+478645us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400012cd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:02.765883 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3060061952}
I20260806 20:36:02.678179 14469 raft_consensus.cc:2379] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:36:02.650864 17478 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:02.747095 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1614) took 1439 ms. Trace:
I20260806 20:36:02.727646 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1613) took 1420 ms. Trace:
I20260806 20:36:03.073803 14631 rpcz_store.cc:276] 0806 20:36:01.307254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.162036 (+854782us) service_pool.cc:227] Handling call
0806 20:36:02.165478 (+ 3442us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.203632 (+ 38154us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.203676 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:02.203729 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:02.203780 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.727344 (+523564us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b54002c2254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:02.727408 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.727576 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87258880}
I20260806 20:36:02.639645 17255 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:02.875015 17578 log.cc:924] Time spent T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.466s user 0.001s sys 0.000s
W20260806 20:36:02.874848 17582 log.cc:924] Time spent T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.501s user 0.000s sys 0.001s
I20260806 20:36:03.014211 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:03.024739 14630 raft_consensus.cc:2396] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 20:36:02.695628 15178 pending_rounds.cc:76] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Aborting all ops after (but not including) 5
W20260806 20:36:02.810913 17581 log.cc:924] Time spent T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.432s user 0.001s sys 0.001s
I20260806 20:36:03.028942 14595 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:03.097246 15178 pending_rounds.cc:97] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260806 20:36:03.097633 15178 raft_consensus.cc:2866] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:36:03.098035 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 69220us
I20260806 20:36:03.099588 14595 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:02.663952 14441 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.613s user 0.000s sys 0.014s
W20260806 20:36:03.100704 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 1.04956e+06us
I20260806 20:36:02.668469 17557 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:02.769610 17576 log.cc:924] Time spent T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.381s user 0.002s sys 0.000s
I20260806 20:36:02.665894 17403 raft_consensus.cc:482] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:02.975023 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:03.108106 14547 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:03.109587 17602 raft_consensus.cc:2726] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:02.974097 15151 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:03.111200 17332 consensus_queue.cc:1035] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 20:36:03.111788 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 137857us
I20260806 20:36:03.112864 15151 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:03.112675 17599 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:36:03.114061 15151 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:03.114538 17403 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:03.115176 15151 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:02.972401 14626 rpcz_store.cc:276] 0806 20:36:01.269572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.112653 (+843081us) service_pool.cc:227] Handling call
0806 20:36:02.113152 (+ 499us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.113612 (+ 460us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.113654 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.114201 (+ 547us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.375209 (+261008us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:02.375253 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.375318 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:02.608912 (+233594us) raft_consensus.cc:1671] finished
0806 20:36:02.608933 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.609047 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3073,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":84,"thread_start_us":260560,"threads_started":1,"wal-append.queue_time_us":363311}
I20260806 20:36:03.116752 17577 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:03.116928 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.117496 17577 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:03.118045 17577 raft_consensus.cc:686] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:03.118772 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.119002 17577 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.677475 17546 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:03.045969 14473 raft_consensus.cc:1226] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: []
W20260806 20:36:02.683913 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 388964us
I20260806 20:36:02.661572 17465 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:02.755435 17574 log.cc:924] Time spent T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.373s user 0.000s sys 0.001s
I20260806 20:36:03.047169 14521 leader_election.cc:398] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:36:03.121357 14521 leader_election.cc:401] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260806 20:36:03.121924 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 74961.2us
I20260806 20:36:03.033650 17523 raft_consensus.cc:2726] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:36:03.122601 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.123034 17523 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209) with higher term. Message: Invalid argument: T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:36:03.039347 17591 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:03.124022 17591 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:02.974602 17537 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260806 20:36:03.124879 17269 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:03.125675 17537 raft_consensus.cc:3032] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:36:03.126111 17537 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:03.127171 17537 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.695196 17455 consensus_queue.cc:237] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:03.128436 17593 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.201s user 0.001s sys 0.001s
I20260806 20:36:03.053840 15184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 648) took 2247 ms. Trace:
I20260806 20:36:02.741341 17579 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:02.771426 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1595) took 3228 ms. Trace:
I20260806 20:36:03.164031 14632 rpcz_store.cc:276] 0806 20:35:59.542981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:35:59.543211 (+ 230us) service_pool.cc:227] Handling call
0806 20:35:59.559843 (+ 16632us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.118250 (+2558407us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.771156 (+652906us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b5400313154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:02.771340 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5882654592}
I20260806 20:36:03.067392 14596 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:03.165548 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 98235.8us
I20260806 20:36:03.065218 17469 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:03.064769 15183 rpcz_store.cc:276] 0806 20:36:00.760770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.821941 (+ 61171us) service_pool.cc:227] Handling call
0806 20:36:02.766553 (+1944612us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400710554. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:02.766723 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2316758912}
I20260806 20:36:02.967880 17614 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:03.175096 17614 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.713394 17590 raft_consensus.cc:2781] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:03.176817 17590 raft_consensus.cc:686] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:03.177489 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.178206 14632 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:03.177744 17590 consensus_queue.cc:237] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:03.069746 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:03.179113 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.179160 14548 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:03.180495 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:03.180281 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:02.669543 17530 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:02.970182 17486 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:02.695120 17302 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:03.084824 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 455754us
W20260806 20:36:03.036386 17531 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.003s sys 0.007s
I20260806 20:36:03.089344 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.075623 14629 rpcz_store.cc:276] 0806 20:36:01.307592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.187429 (+879837us) service_pool.cc:227] Handling call
0806 20:36:02.187809 (+ 380us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.188013 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:36:02.434841 (+246828us) log.cc:842] Serialized 14 byte log entry
0806 20:36:02.746423 (+311582us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b1800486040. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b318282c 00007fb9b05ad9dc 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05cae22 00007fb9b31745bb 00007fb9b317a063 00007fb9b317a57b
0806 20:36:02.746652 (+ 229us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:02.746713 (+ 61us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:02.746776 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.746864 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:02.747023 (+ 159us) inbound_call.cc:173] Queueing success response
0806 20:36:03.035443 (+288420us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
Metrics: {"mutex_wait_us":31242,"prepare.queue_time_us":236085,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":3975,"raft.queue_time_us":2000,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"spinlock_wait_cycles":315477632,"thread_start_us":306259,"threads_started":2,"wal-append.queue_time_us":287372}
W20260806 20:36:03.101400 14441 leader_election.cc:334] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399): Timed out: RequestConsensusVote RPC to 127.12.248.132:42399 timed out after 1.925s (SENT)
W20260806 20:36:03.200023 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 98658.6us
I20260806 20:36:03.096186 17542 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260806 20:36:03.103992 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1794) took 2646 ms. Trace:
I20260806 20:36:03.221342 14553 rpcz_store.cc:276] 0806 20:36:00.457206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.219697 (+1762491us) service_pool.cc:227] Handling call
0806 20:36:02.225830 (+ 6133us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.250921 (+ 25091us) raft_consensus.cc:1520] Early marking committed up to index 1
0806 20:36:02.250969 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.294229 (+ 43260us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.613071 (+318842us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:36:02.613112 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.613184 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:03.103706 (+490522us) raft_consensus.cc:1671] finished
0806 20:36:03.103730 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.103906 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7629,"raft.queue_time_us":27260,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":6588,"thread_start_us":280358,"threads_started":1,"wal-append.queue_time_us":473708}
I20260806 20:36:03.105253 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.106700 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.111917 17160 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:36:02.733269 15182 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 503) took 2003 ms (client timeout 1898 ms). Trace:
I20260806 20:36:02.926146 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:03.122596 14521 leader_election.cc:302] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:03.124045 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:36:03.238600 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 116103us
I20260806 20:36:03.238656 14630 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:03.119545 14626 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:03.156317 14595 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:03.157915 15151 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:03.087059 15181 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.004s sys 0.050s
I20260806 20:36:03.121302 14442 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:36:03.246021 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 124825us
I20260806 20:36:03.246073 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.166939 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1793) took 2754 ms. Trace:
I20260806 20:36:03.174028 14596 leader_election.cc:302] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:03.174531 17269 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:03.168233 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1684) took 3589 ms. Trace:
I20260806 20:36:03.248783 14551 rpcz_store.cc:276] 0806 20:35:59.578760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.116032 (+2537272us) service_pool.cc:227] Handling call
0806 20:36:02.116508 (+ 476us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.119265 (+ 2757us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.119309 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.119765 (+ 456us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.408444 (+288679us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:02.408517 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.408588 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:03.167960 (+759372us) raft_consensus.cc:1671] finished
0806 20:36:03.167991 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.168160 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":288229,"threads_started":1,"wal-append.queue_time_us":379657}
I20260806 20:36:03.248528 14554 rpcz_store.cc:276] 0806 20:36:00.412851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.143637 (+1730786us) service_pool.cc:227] Handling call
0806 20:36:02.143946 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.144194 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:02.144221 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.144735 (+ 514us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.494292 (+349557us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:02.494420 (+ 128us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.494621 (+ 201us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:03.127780 (+633159us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:03.129837 (+ 2057us) raft_consensus.cc:1671] finished
0806 20:36:03.129860 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.166845 (+ 36985us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57891584,"thread_start_us":348953,"threads_started":1,"wal-append.queue_time_us":782148}
I20260806 20:36:03.252214 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1692) took 2547 ms. Trace:
I20260806 20:36:03.181375 14548 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:03.179862 17478 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:02.774225 17555 raft_consensus.cc:3032] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:36:03.112725 17332 raft_consensus.cc:2726] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:03.130105 15184 rpcz_store.cc:276] 0806 20:36:00.806030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.819198 (+ 13168us) service_pool.cc:227] Handling call
0806 20:36:00.819592 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:00.820031 (+ 439us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:00.820059 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:01.101939 (+281880us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.398102 (+1296163us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:02.398139 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.398201 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:02.771209 (+373008us) spinlock_profiling.cc:243] Waited 2.91 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:03.053489 (+282280us) raft_consensus.cc:1671] finished
0806 20:36:03.053512 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.053752 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":39492,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":46973,"spinlock_wait_cycles":6686976,"thread_start_us":1295776,"threads_started":1,"wal-append.queue_time_us":1270604}
I20260806 20:36:03.182689 17529 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:03.265470 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 91516.2us
I20260806 20:36:03.263710 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1608) took 2085 ms. Trace:
I20260806 20:36:03.232865 14547 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
W20260806 20:36:03.234714 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 121117us
W20260806 20:36:03.241792 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 84172.2us
I20260806 20:36:03.241894 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:03.239428 14521 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:03.243433 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 87319.6us
W20260806 20:36:03.374748 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 135469us
I20260806 20:36:03.168143 17602 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.177s
I20260806 20:36:03.255023 14442 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:03.202136 17531 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.979s user 0.004s sys 0.007s
I20260806 20:36:03.191942 17537 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:03.252710 14555 rpcz_store.cc:276] 0806 20:36:00.704484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.422061 (+1717577us) service_pool.cc:227] Handling call
0806 20:36:02.479021 (+ 56960us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.479218 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:36:03.077583 (+598365us) log.cc:842] Serialized 14 byte log entry
0806 20:36:03.251945 (+174362us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:03.251999 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:03.252031 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.252074 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.252170 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":259,"thread_start_us":642308,"threads_started":2}
W20260806 20:36:03.258607 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 60453.9us
W20260806 20:36:03.259763 14441 connection.cc:232] RPC call timeout handler was delayed by -1.0862205079756677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
I20260806 20:36:03.389827 14441 leader_election.cc:398] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6.
I20260806 20:36:03.270920 14551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c"
dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
mode: GRACEFUL
new_leader_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
from {username='slave'} at 127.0.0.1:48150
I20260806 20:36:03.390969 14595 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:03.391505 14551 raft_consensus.cc:595] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Received request to transfer leadership to TS df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:03.392046 14551 raft_consensus.cc:603] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:36:03.333598 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 53017.4us
I20260806 20:36:03.266263 14628 rpcz_store.cc:276] 0806 20:36:01.177914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:01.178069 (+ 155us) service_pool.cc:227] Handling call
0806 20:36:01.184690 (+ 6621us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:01.208680 (+ 23990us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:01.208713 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:01.269006 (+ 60293us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.047199 (+778193us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:02.047251 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.047341 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:03.263551 (+1216210us) raft_consensus.cc:1671] finished
0806 20:36:03.263568 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.263665 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23510,"raft.queue_time_us":23650,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":57538,"thread_start_us":738497,"threads_started":1,"wal-append.queue_time_us":1012052}
I20260806 20:36:03.403258 14595 leader_election.cc:398] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:36:03.403687 14595 leader_election.cc:401] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:36:03.372795 17529 raft_consensus.cc:2781] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:36:03.245615 15181 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s user 0.004s sys 0.052s
I20260806 20:36:03.354466 14631 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:03.262729 17546 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:03.275141 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1693) took 2484 ms. Trace:
I20260806 20:36:03.337018 15151 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:03.371984 14630 raft_consensus.cc:2454] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:03.256151 17465 raft_consensus.cc:2781] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:03.258873 17457 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:36:03.255468 17269 raft_consensus.cc:482] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:03.341055 14553 raft_consensus.cc:2379] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:03.260506 17555 raft_consensus.cc:729] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:03.378844 14596 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:03.254837 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.191419 17579 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:03.380724 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 125801us
W20260806 20:36:03.237473 15182 rpcz_store.cc:269] 0806 20:36:00.729347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:00.753120 (+ 23773us) service_pool.cc:227] Handling call
0806 20:36:02.732968 (+1979848us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54004d08d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:02.733178 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2316568448}
I20260806 20:36:03.395576 17580 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:03.210353 17473 consensus_queue.cc:237] T 6b370f85a8b944198972759bc1eb8061 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:03.417596 14548 rpcz_store.cc:276] 0806 20:36:00.791072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.253492 (+2462420us) service_pool.cc:227] Handling call
0806 20:36:03.274427 (+ 20935us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:03.274699 (+ 272us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:03.274737 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:03.274795 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:03.274851 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.274919 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.275061 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:03.210038 17491 raft_consensus.cc:2726] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:03.378052 17523 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:36:03.432968 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.434689 13282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:36:03.374153 17614 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:03.395359 17543 consensus_meta.cc:287] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.961s user 0.002s sys 0.009s
I20260806 20:36:03.433333 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.419893 17599 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:03.438061 14442 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:03.438261 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.438908 14551 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:03.394970 17458 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:36:03.377480 17332 consensus_queue.cc:1046] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260806 20:36:03.454336 14518 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:03.420117 17155 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:03.549036 13282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 20:36:03.556401 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 153300us
I20260806 20:36:03.436036 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 660) took 2394 ms. Trace:
I20260806 20:36:03.653640 14595 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:03.488870 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 110102us
I20260806 20:36:03.417368 17546 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:03.388012 17589 consensus_queue.cc:1046] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.104s
I20260806 20:36:03.408499 17376 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.084s
I20260806 20:36:03.573825 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1696) took 2742 ms. Trace:
W20260806 20:36:03.573827 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 135854us
I20260806 20:36:03.684283 14551 rpcz_store.cc:276] 0806 20:36:00.831092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.407252 (+2576160us) service_pool.cc:227] Handling call
0806 20:36:03.438684 (+ 31432us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:03.573542 (+134858us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.573755 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:03.684792 14442 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:03.685838 17613 raft_consensus.cc:2726] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:36:03.686193 14442 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:03.687218 17613 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:36:03.687810 17613 raft_consensus.cc:686] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:03.433984 17235 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:03.688766 17235 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:03.690366 17352 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 20:36:03.688226 17546 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:03.691285 17235 leader_election.cc:288] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:03.443687 17269 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:03.688974 17613 consensus_queue.cc:237] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:03.674942 14479 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.051s user 0.028s sys 0.075s
W20260806 20:36:03.694082 14479 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.085s user 0.028s sys 0.075s
I20260806 20:36:03.546586 17523 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:36:03.701152 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.026s user 0.019s sys 0.014s
I20260806 20:36:03.439042 17465 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:03.556681 17605 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:03.574256 17608 log.cc:924] Time spent T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.259s user 0.002s sys 0.000s
I20260806 20:36:03.443529 17457 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:36:03.723125 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 285619us
I20260806 20:36:03.556980 17137 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:36:03.653450 15187 rpcz_store.cc:276] 0806 20:36:01.042020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.251804 (+1209784us) service_pool.cc:227] Handling call
0806 20:36:02.733557 (+481753us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.733843 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:36:02.900789 (+166946us) log.cc:842] Serialized 14 byte log entry
0806 20:36:03.160340 (+259551us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:03.160395 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:03.160473 (+ 78us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.161067 (+ 594us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:03.410543 (+249476us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b180046f7e0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b0598f96 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684
0806 20:36:03.435846 (+ 25303us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54004d08d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:03.435974 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2397,"prepare.queue_time_us":368212,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":102,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":18742,"spinlock_wait_cycles":1109338752,"thread_start_us":425585,"threads_started":2}
I20260806 20:36:03.547252 17596 raft_consensus.cc:2726] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:03.595212 17542 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:36:03.726387 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.051s user 0.019s sys 0.018s
W20260806 20:36:03.729759 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 393096us
I20260806 20:36:03.817463 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:03.818365 15183 raft_consensus.cc:3037] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:03.818543 15180 raft_consensus.cc:1261] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:36:03.819824 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:03.820897 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:03.821061 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:03.822587 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:03.822685 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:03.823864 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:03.824525 15186 raft_consensus.cc:3037] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:03.754252 17546 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:03.565929 17255 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:03.729666 14518 leader_election.cc:398] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
W20260806 20:36:03.664805 17576 log.cc:924] Time spent T d2ced25ce9f1484081a8671550489f9b P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20260806 20:36:03.577877 17458 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:03.839139 15185 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
W20260806 20:36:03.734404 17620 log.cc:924] Time spent T e391f934d63c4ad788870d7abc39a79c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.607s user 0.002s sys 0.000s
W20260806 20:36:03.727867 17393 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.399s user 0.011s sys 0.003s
W20260806 20:36:03.877637 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 460958us
I20260806 20:36:03.920585 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.862175 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:03.727665 17566 raft_consensus.cc:2741] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 20:36:03.736740 17465 raft_consensus.cc:3037] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:03.787307 17302 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:03.725544 17523 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:03.877842 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:03.891578 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.597990 17614 consensus_queue.cc:1046] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.259s
I20260806 20:36:03.759096 17457 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:03.825273 17403 consensus_meta.cc:287] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.710s user 0.007s sys 0.004s
W20260806 20:36:03.893317 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 163791us
I20260806 20:36:03.789317 17542 raft_consensus.cc:482] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:03.825217 17602 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election won for term 8
W20260806 20:36:03.983940 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 296081us
I20260806 20:36:03.922560 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:36:03.863135 17617 log.cc:924] Time spent T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.681s user 0.001s sys 0.000s
I20260806 20:36:03.759155 17469 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:03.722532 17160 raft_consensus.cc:2726] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:36:03.962597 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.704357 17155 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:03.885257 17530 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:03.907220 17458 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:03.926229 17455 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260806 20:36:03.967127 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:03.967031 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:03.968502 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:36:03.933920 17610 log.cc:924] Time spent T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.702s user 0.001s sys 0.001s
I20260806 20:36:03.918267 17527 raft_consensus.cc:2726] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:03.898336 17557 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:03.568756 17543 raft_consensus.cc:504] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:03.920635 17478 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:03.989169 14518 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:03.853345 17269 raft_consensus.cc:504] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:03.933465 17393 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:04.022629 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 367372us
I20260806 20:36:04.155333 14596 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:04.001087 14442 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:04.066376 17635 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.187s user 0.000s sys 0.000s
I20260806 20:36:04.062466 15151 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:04.121383 15185 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:04.196841 14631 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:36:04.197974 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:04.198714 17517 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:04.199470 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:04.199395 17517 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.023015 15184 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:04.201395 17602 raft_consensus.cc:686] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:04.202593 17602 consensus_queue.cc:237] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.203387 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:04.204171 15184 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:03.788810 17605 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:03.730559 17555 consensus_queue.cc:260] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.217123 17650 raft_consensus.cc:482] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:04.217258 17555 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:36:04.217679 17650 raft_consensus.cc:504] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.221050 15184 raft_consensus.cc:2354] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
W20260806 20:36:04.223047 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 1.05355e+06us
I20260806 20:36:04.223040 17605 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:36:04.229621 17604 log.cc:924] Time spent T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 1.327s user 0.000s sys 0.015s
W20260806 20:36:04.065735 17624 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.629s user 0.000s sys 0.001s
W20260806 20:36:04.232229 17612 log.cc:924] Time spent T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 1.305s user 0.002s sys 0.002s
I20260806 20:36:03.983513 17542 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:04.087306 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 433726us
W20260806 20:36:04.067077 17616 log.cc:924] Time spent T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
W20260806 20:36:04.064126 17615 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.577s user 0.001s sys 0.000s
W20260806 20:36:04.158771 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 157776us
I20260806 20:36:04.252770 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1619) took 2208 ms. Trace:
I20260806 20:36:04.253252 14627 rpcz_store.cc:276] 0806 20:36:02.044106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.195398 (+151292us) service_pool.cc:227] Handling call
0806 20:36:02.195735 (+ 337us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.195936 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.195960 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.196359 (+ 399us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.628643 (+432284us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:02.629433 (+ 790us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.629524 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:04.231501 (+1601977us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:04.234453 (+ 2952us) raft_consensus.cc:1671] finished
0806 20:36:04.234490 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.252691 (+ 18201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59337088,"thread_start_us":431485,"threads_started":1,"wal-append.queue_time_us":715929}
I20260806 20:36:04.145115 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:04.261945 14477 raft_consensus.cc:1226] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:04.161571 17269 leader_election.cc:288] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:04.162101 17393 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:04.162607 14632 raft_consensus.cc:1226] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
W20260806 20:36:04.162838 17537 consensus_meta.cc:287] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.768s user 0.008s sys 0.000s
I20260806 20:36:04.163870 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1800) took 3628 ms. Trace:
I20260806 20:36:04.279145 14552 rpcz_store.cc:276] 0806 20:36:00.535294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.377916 (+1842622us) service_pool.cc:227] Handling call
0806 20:36:02.391578 (+ 13662us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.391744 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.391758 (+ 14us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.392114 (+ 356us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.610166 (+218052us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:02.610213 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.610298 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:04.163644 (+1553346us) raft_consensus.cc:1671] finished
0806 20:36:04.163670 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.163805 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":183478,"threads_started":1,"wal-append.queue_time_us":656627}
I20260806 20:36:04.168933 17546 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:04.171687 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1635) took 1108 ms. Trace:
I20260806 20:36:04.181205 17579 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:04.134254 17462 consensus_meta.cc:287] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.760s user 0.015s sys 0.000s
I20260806 20:36:04.195331 17656 raft_consensus.cc:482] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:04.065336 17634 log.cc:924] Time spent T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.336s user 0.000s sys 0.000s
W20260806 20:36:04.064761 17631 log.cc:924] Time spent T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.410s user 0.001s sys 0.000s
I20260806 20:36:04.063282 17614 raft_consensus.cc:3032] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:04.129525 15186 raft_consensus.cc:2454] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:04.206007 16844 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:04.209056 17536 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:36:04.211462 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 149091us
I20260806 20:36:04.224385 14444 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:04.119100 15187 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:04.235067 14595 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:04.394194 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 159241us
W20260806 20:36:04.241994 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 252951us
I20260806 20:36:04.246291 15183 raft_consensus.cc:2454] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:04.244964 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1695) took 3439 ms. Trace:
W20260806 20:36:04.252135 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 96890.5us
W20260806 20:36:04.396476 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 172136us
W20260806 20:36:04.258035 14630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.010s sys 0.031s
I20260806 20:36:04.220243 17650 leader_election.cc:288] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:04.270965 14479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.089s sys 0.123s
W20260806 20:36:04.397033 14630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s user 0.010s sys 0.032s
I20260806 20:36:04.145691 17589 raft_consensus.cc:482] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:04.152539 17531 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
W20260806 20:36:04.260694 17617 log.cc:924] Time spent T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20260806 20:36:04.236351 17557 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:04.290699 14626 rpcz_store.cc:276] 0806 20:36:03.063202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.255049 (+191847us) service_pool.cc:227] Handling call
0806 20:36:03.255367 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:03.256596 (+ 1229us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:03.256622 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:03.278311 (+ 21689us) log.cc:842] Serialized 26 byte log entry
0806 20:36:03.545073 (+266762us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:03.545129 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.545230 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:04.171470 (+626240us) raft_consensus.cc:1671] finished
0806 20:36:04.171492 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.171626 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1422,"raft.queue_time_us":1301,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":11928,"thread_start_us":131671,"threads_started":1,"wal-append.queue_time_us":497381}
I20260806 20:36:03.990005 17403 raft_consensus.cc:504] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.021938 17600 consensus_queue.cc:1046] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.247s
W20260806 20:36:04.325269 14624 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.943s user 0.016s sys 0.016s
W20260806 20:36:04.417362 14624 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s user 0.016s sys 0.016s
I20260806 20:36:04.364050 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:04.268102 17332 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:04.367743 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:04.419838 14552 raft_consensus.cc:2379] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 4: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:36:04.373476 17587 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.385s user 0.005s sys 0.010s
I20260806 20:36:04.376905 17580 raft_consensus.cc:482] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:04.376622 17645 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.115s
I20260806 20:36:04.377462 17656 raft_consensus.cc:504] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:04.378571 17637 log.cc:924] Time spent T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.399s user 0.001s sys 0.000s
I20260806 20:36:04.363870 17462 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.380074 15181 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 650) took 3560 ms. Trace:
I20260806 20:36:04.402464 17641 raft_consensus.cc:2726] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:36:04.452834 15181 rpcz_store.cc:276] 0806 20:36:00.819530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.195719 (+1376189us) service_pool.cc:227] Handling call
0806 20:36:02.196045 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.307015 (+110970us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.311828 (+ 4813us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.378474 (+ 66646us) log.cc:842] Serialized 26 byte log entry
0806 20:36:03.445294 (+1066820us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:03.453206 (+ 7912us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:04.228412 (+775206us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b540045ea54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:04.228489 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:04.379647 (+151158us) raft_consensus.cc:1671] finished
0806 20:36:04.379683 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.379997 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148864384,"thread_start_us":988530,"threads_started":1,"wal-append.queue_time_us":930121}
I20260806 20:36:04.382335 15180 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
W20260806 20:36:04.397307 14479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.090s sys 0.123s
I20260806 20:36:04.158738 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:04.395844 14555 rpcz_store.cc:276] 0806 20:36:00.805180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.388582 (+2583402us) service_pool.cc:227] Handling call
0806 20:36:03.395074 (+ 6492us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:03.411976 (+ 16902us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:36:03.416075 (+ 4099us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:03.447179 (+ 31104us) log.cc:842] Serialized 26 byte log entry
0806 20:36:03.923373 (+476194us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:03.923425 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.923532 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:04.244695 (+321163us) raft_consensus.cc:1671] finished
0806 20:36:04.244719 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.244886 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11679,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":33109,"thread_start_us":325962,"threads_started":1,"wal-append.queue_time_us":676665}
I20260806 20:36:04.399353 15185 raft_consensus.cc:1261] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:36:04.400640 14596 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:04.401695 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1638) took 1237 ms. Trace:
I20260806 20:36:04.319715 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1637) took 1156 ms. Trace:
I20260806 20:36:04.367769 17614 raft_consensus.cc:729] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:04.411731 14518 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:04.423424 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1687) took 3852 ms. Trace:
I20260806 20:36:04.457931 14632 rpcz_store.cc:276] 0806 20:36:03.163587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.493515 (+329928us) service_pool.cc:227] Handling call
0806 20:36:03.549141 (+ 55626us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:04.278645 (+729504us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:04.319387 (+ 40742us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b540039f3d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:04.319633 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1410314496}
I20260806 20:36:04.458479 14549 rpcz_store.cc:276] 0806 20:36:00.570596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.398301 (+1827705us) service_pool.cc:227] Handling call
0806 20:36:02.411355 (+ 13054us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.427192 (+ 15837us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.427229 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.427662 (+ 433us) log.cc:842] Serialized 26 byte log entry
0806 20:36:02.800802 (+373140us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:02.800859 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:02.800959 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:04.423203 (+1622244us) raft_consensus.cc:1671] finished
0806 20:36:04.423224 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.423362 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":300254,"threads_started":1,"wal-append.queue_time_us":1187856}
I20260806 20:36:04.459444 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:04.460494 14595 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:04.457808 14630 rpcz_store.cc:276] 0806 20:36:03.164258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.576441 (+412183us) service_pool.cc:227] Handling call
0806 20:36:03.662952 (+ 86511us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:03.663192 (+ 240us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:03.757495 (+ 94303us) log.cc:842] Serialized 14 byte log entry
0806 20:36:04.401361 (+643866us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:04.401412 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:04.401465 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:04.401530 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.401639 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8172,"prepare.queue_time_us":123763,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":13738,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"thread_start_us":639935,"threads_started":1}
W20260806 20:36:04.406215 17591 consensus_meta.cc:287] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.282s user 0.012s sys 0.001s
I20260806 20:36:04.421578 17332 raft_consensus.cc:482] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:04.439646 17580 raft_consensus.cc:504] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.397030 17536 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:04.451679 14624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1627) took 1822 ms. Trace:
I20260806 20:36:04.481884 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:04.482723 14549 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:04.483443 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:04.488193 14632 raft_consensus.cc:1226] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:04.452375 17656 leader_election.cc:288] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:04.510303 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:04.511185 14470 raft_consensus.cc:2454] T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:04.408599 17505 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:04.467331 17403 leader_election.cc:288] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:04.408846 17650 raft_consensus.cc:482] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:36:04.423317 17587 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.522s user 0.005s sys 0.011s
W20260806 20:36:04.021587 17611 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.663s user 0.000s sys 0.001s
I20260806 20:36:03.971412 17235 consensus_queue.cc:1046] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.504s
I20260806 20:36:04.413378 15178 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 662) took 1932 ms. Trace:
W20260806 20:36:04.477792 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 77256.1us
I20260806 20:36:04.490860 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1682) took 2074 ms. Trace:
I20260806 20:36:04.408339 17393 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260806 20:36:04.469420 17455 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:04.412424 17589 raft_consensus.cc:504] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.412876 17531 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.441021 17557 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.304054 17457 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.668375 14595 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:36:04.610085 14549 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 8.
I20260806 20:36:04.467940 15182 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:04.595332 14470 raft_consensus.cc:1226] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260806 20:36:04.566126 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:04.626562 14479 rpcz_store.cc:276] 0806 20:36:02.416647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.420757 (+ 4110us) service_pool.cc:227] Handling call
0806 20:36:02.479921 (+ 59164us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:02.480164 (+ 243us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:03.388361 (+908197us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b1800455200. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb638 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:36:03.694535 (+306174us) log.cc:842] Serialized 14 byte log entry
0806 20:36:04.469229 (+774694us) log.cc:842] Serialized 14 byte log entry
0806 20:36:04.490428 (+ 21199us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:04.490494 (+ 66us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:04.490558 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:04.490652 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.490788 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28897,"prepare.queue_time_us":132515,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":5550,"spinlock_wait_cycles":30705536,"thread_start_us":1845121,"threads_started":2}
I20260806 20:36:04.673087 14474 raft_consensus.cc:2396] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:36:04.673774 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:04.476758 17332 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:04.569918 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:04.493907 17536 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:04.571064 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:04.573534 17523 consensus_meta.cc:287] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.605s user 0.005s sys 0.005s
I20260806 20:36:04.478055 17591 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.675163 14549 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:36:04.717013 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:04.718317 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:04.719062 14549 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
W20260806 20:36:04.569847 17137 consensus_meta.cc:287] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.783s user 0.010s sys 0.000s
I20260806 20:36:04.720755 17137 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:36:04.627246 15178 rpcz_store.cc:276] 0806 20:36:02.480991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.489561 (+ 8570us) service_pool.cc:227] Handling call
0806 20:36:02.651353 (+161792us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:36:03.098473 (+447120us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:03.098544 (+ 71us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:36:03.099191 (+ 647us) log.cc:842] Serialized 52 byte log entry
0806 20:36:03.209304 (+110113us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:03.209374 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.237016 (+ 27642us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:04.389670 (+1152654us) raft_consensus.cc:1671] finished
0806 20:36:04.395798 (+ 6128us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.409282 (+ 13484us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4885,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":49601,"thread_start_us":92011,"threads_started":1,"wal-append.queue_time_us":309506}
I20260806 20:36:04.630934 17155 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:04.664583 17650 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.724716 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:04.725442 17462 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:04.725981 17462 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:04.720026 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:04.564754 17486 consensus_meta.cc:287] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.363s user 0.013s sys 0.000s
I20260806 20:36:04.565167 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:04.626849 14596 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:04.695094 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:04.629695 17455 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:04.564819 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:04.714424 14555 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:36:04.631378 17376 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:04.524498 17649 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:04.524426 17646 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:04.720108 15179 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:04.727077 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:04.727679 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:04.732872 17611 log.cc:924] Time spent T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260806 20:36:04.732028 15183 raft_consensus.cc:2396] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20260806 20:36:04.734731 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 107970us
I20260806 20:36:04.735327 14475 raft_consensus.cc:2379] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:36:04.738444 14546 raft_consensus.cc:1261] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260806 20:36:04.740703 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:04.521974 17655 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.205s
I20260806 20:36:04.800776 17557 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:04.804486 17655 consensus_queue.cc:1225] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer 92828dd6dbe74f29a04d4e702b502621 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 20:36:04.567324 17580 leader_election.cc:288] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:04.742296 14629 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.292s user 0.005s sys 0.038s
W20260806 20:36:04.806321 14629 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.372s user 0.006s sys 0.039s
I20260806 20:36:04.726440 17650 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:04.807282 17650 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:36:04.728633 14552 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:04.812700 17638 raft_consensus.cc:3032] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:36:04.813248 17638 raft_consensus.cc:729] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:04.814298 17638 consensus_queue.cc:260] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.815701 17638 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:04.819064 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1623) took 2321 ms. Trace:
I20260806 20:36:04.820544 14546 pending_rounds.cc:76] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 3
I20260806 20:36:04.820816 14546 pending_rounds.cc:97] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:36:04.821056 14546 raft_consensus.cc:2866] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:36:04.748253 17302 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:04.822075 17302 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.825132 14549 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:04.826009 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:04.826727 15179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 663) took 1006 ms. Trace:
I20260806 20:36:04.827155 15186 raft_consensus.cc:1261] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:36:04.827164 15179 rpcz_store.cc:276] 0806 20:36:03.820299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.822266 (+ 1967us) service_pool.cc:227] Handling call
0806 20:36:04.826569 (+1004303us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b540037f0d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:04.826673 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2062124800}
W20260806 20:36:04.610334 17465 consensus_meta.cc:287] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.639s user 0.012s sys 0.000s
I20260806 20:36:04.667449 17656 raft_consensus.cc:2726] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260806 20:36:04.850903 14547 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.435s user 0.015s sys 0.030s
W20260806 20:36:04.851243 14547 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.435s user 0.015s sys 0.030s
I20260806 20:36:04.738596 17393 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:04.793691 17591 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:04.736662 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:04.796780 17255 raft_consensus.cc:3032] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:36:04.806170 17457 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:04.806568 17630 log.cc:924] Time spent T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 1.314s user 0.004s sys 0.002s
I20260806 20:36:04.726430 17462 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.565614 17652 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:36:04.872105 14548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.011s sys 0.009s
I20260806 20:36:04.744812 17649 raft_consensus.cc:482] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:04.748118 17646 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:36:04.664150 17473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.017s sys 0.017s
W20260806 20:36:04.794023 17458 consensus_meta.cc:287] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.648s user 0.009s sys 0.000s
W20260806 20:36:04.738924 14550 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.609s user 0.000s sys 0.039s
W20260806 20:36:04.905742 14550 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.776s user 0.001s sys 0.039s
W20260806 20:36:04.858053 14441 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.003s user 0.000s sys 0.083s
W20260806 20:36:04.906780 14441 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.052s user 0.000s sys 0.084s
W20260806 20:36:04.907217 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 1.0527e+06us
W20260806 20:36:04.735733 17577 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.041s sys 0.114s
I20260806 20:36:04.729097 17523 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.832690 14552 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:36:04.819485 14629 rpcz_store.cc:276] 0806 20:36:02.497478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.202298 (+704820us) service_pool.cc:227] Handling call
0806 20:36:03.202651 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:03.202857 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:36:03.264663 (+ 61806us) log.cc:842] Serialized 14 byte log entry
0806 20:36:04.818715 (+1554052us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:04.818766 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:04.818813 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:04.818880 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.819008 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86066,"prepare.queue_time_us":178730,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":5727,"raft.queue_time_us":235158,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":93,"thread_start_us":1535686,"threads_started":2,"wal-append.queue_time_us":898957}
I20260806 20:36:04.870067 17393 raft_consensus.cc:686] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:04.867123 17255 raft_consensus.cc:729] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
W20260806 20:36:04.898314 14521 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.625s user 0.019s sys 0.050s
I20260806 20:36:04.843290 17486 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:04.826609 14549 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:36:04.735834 17455 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:04.977298 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 419) took 1106 ms. Trace:
I20260806 20:36:04.977792 14549 rpcz_store.cc:276] 0806 20:36:03.871048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.825706 (+954658us) service_pool.cc:227] Handling call
0806 20:36:04.947915 (+122209us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:04.904881 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 495156us
I20260806 20:36:04.564976 17614 consensus_queue.cc:260] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.672019 17579 consensus_queue.cc:1035] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.184s
W20260806 20:36:04.872852 14548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s user 0.012s sys 0.009s
W20260806 20:36:04.948055 14521 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.675s user 0.019s sys 0.050s
W20260806 20:36:04.910418 17473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s user 0.018s sys 0.018s
I20260806 20:36:05.034140 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:05.035117 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.040268 14444 leader_election.cc:302] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:04.950269 14519 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.018s sys 0.056s
W20260806 20:36:05.042271 14519 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.764s user 0.019s sys 0.056s
I20260806 20:36:04.905047 17649 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:04.855254 17465 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.908993 17646 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:36:04.909431 17480 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.007s sys 0.023s
I20260806 20:36:05.028848 15151 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:04.912658 17577 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.941s user 0.041s sys 0.114s
I20260806 20:36:04.929035 14441 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.042641 14444 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:36:04.906445 17557 raft_consensus.cc:482] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:05.055512 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:05.117944 14632 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:05.117456 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 74905.6us
I20260806 20:36:05.056330 14631 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260806 20:36:05.075587 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.116575 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:04.913197 17458 raft_consensus.cc:504] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:04.746737 17376 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.154851 14549 raft_consensus.cc:2379] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 5: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:36:05.153899 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 224945us
I20260806 20:36:05.166538 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 424) took 1276 ms. Trace:
I20260806 20:36:05.060695 17589 leader_election.cc:288] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:05.174824 17465 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:05.166759 14441 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.176226 17523 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 9 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:05.177376 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:05.178531 17462 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:05.181654 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.182869 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:36:05.073204 17670 log.cc:924] Time spent T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260806 20:36:05.054756 17614 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:36:05.144997 17480 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.896s user 0.008s sys 0.026s
I20260806 20:36:05.232404 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.234385 14632 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:36:05.184376 14549 rpcz_store.cc:276] 0806 20:36:03.890172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.978700 (+1088528us) service_pool.cc:227] Handling call
0806 20:36:05.166460 (+187760us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:05.146896 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 118157us
I20260806 20:36:04.979815 17486 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.048085 17667 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:05.183862 14626 raft_consensus.cc:3037] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:05.119867 17517 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:05.229751 17302 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:05.051671 17554 consensus_queue.cc:1159] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" responder_term: 7 status { last_received { term: 6 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:36:05.257632 17457 raft_consensus.cc:2726] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:05.118381 17668 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:05.151397 17258 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:36:05.259967 17457 raft_consensus.cc:2781] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:05.265551 17258 raft_consensus.cc:2726] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:36:05.241505 15151 leader_election.cc:302] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:36:05.244922 14521 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.251539 17667 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.268128 17667 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:05.252312 16844 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:05.060693 17393 consensus_queue.cc:237] T 568833ac14f74d16a95723bbfa721a96 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:04.481699 14624 rpcz_store.cc:276] 0806 20:36:02.634092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.224769 (+590677us) service_pool.cc:227] Handling call
0806 20:36:03.225250 (+ 481us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:03.225555 (+ 305us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:36:03.381814 (+156259us) log.cc:842] Serialized 14 byte log entry
0806 20:36:04.441267 (+1059453us) log.cc:842] Serialized 14 byte log entry
0806 20:36:04.451328 (+ 10061us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:04.451380 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:04.451431 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:04.451496 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:04.451623 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":139507,"prepare.queue_time_us":163003,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":4003,"raft.queue_time_us":47208,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":1035630,"threads_started":1}
I20260806 20:36:05.167408 17557 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.289680 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.275259 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.294410 14632 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:36:05.273610 15151 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.299928 15151 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.304332 14521 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.061003 17255 consensus_queue.cc:260] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:05.249236 17542 consensus_meta.cc:287] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.015s user 0.009s sys 0.005s
W20260806 20:36:05.317926 15184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.010s sys 0.021s
W20260806 20:36:05.318430 15184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s user 0.010s sys 0.021s
W20260806 20:36:05.302582 14553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.014s sys 0.016s
W20260806 20:36:05.319634 14553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.014s sys 0.016s
W20260806 20:36:05.293339 17529 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.726s user 0.010s sys 0.023s
I20260806 20:36:05.269249 17462 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:05.329131 14626 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 9.
I20260806 20:36:05.270835 17517 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:36:05.302675 17659 log.cc:924] Time spent T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
W20260806 20:36:05.302722 17671 log.cc:924] Time spent T 90e4ac7b6cd245aead7963fa7cbc5d2e P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
I20260806 20:36:05.308775 17491 raft_consensus.cc:2781] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:05.336220 14521 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.268710 16844 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:05.344728 14473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.006s sys 0.020s
I20260806 20:36:05.273047 17457 raft_consensus.cc:686] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:05.275127 17557 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:05.378268 15150 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.021s sys 0.049s
W20260806 20:36:05.379001 15150 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.915s user 0.021s sys 0.050s
W20260806 20:36:05.379449 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 915557us
W20260806 20:36:05.317533 17600 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.006s sys 0.031s
I20260806 20:36:05.302173 17302 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:36:05.353178 14476 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.009s sys 0.018s
I20260806 20:36:05.268419 17352 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260806 20:36:05.292197 17332 consensus_meta.cc:287] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.566s user 0.009s sys 0.009s
W20260806 20:36:05.361558 14442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.091s user 0.015s sys 0.015s
I20260806 20:36:05.320374 17667 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:36:05.375327 14473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s user 0.006s sys 0.020s
I20260806 20:36:05.350307 17255 raft_consensus.cc:3037] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:36:05.381454 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 45366.4us
W20260806 20:36:05.383147 14518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.925s user 0.005s sys 0.030s
W20260806 20:36:05.404369 14518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.946s user 0.005s sys 0.031s
I20260806 20:36:05.349357 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:05.394284 14476 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.824s user 0.009s sys 0.018s
I20260806 20:36:05.302414 17258 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:36:05.397665 14442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.127s user 0.015s sys 0.015s
W20260806 20:36:05.332067 17529 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.783s user 0.010s sys 0.023s
I20260806 20:36:05.406381 14549 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:36:05.393900 17600 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.939s user 0.007s sys 0.031s
I20260806 20:36:05.414119 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.362423 17462 raft_consensus.cc:686] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:05.399643 17531 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:36:05.350167 17664 consensus_queue.cc:1159] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" responder_term: 8 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8. Ops: [7.4-7.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:36:05.398607 17667 raft_consensus.cc:482] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:36:05.394098 17602 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.007s sys 0.021s
I20260806 20:36:05.414780 17352 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:05.443374 17235 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.014s sys 0.021s
I20260806 20:36:05.414994 17529 raft_consensus.cc:686] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:05.424906 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:05.425745 17455 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:05.426260 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:36:05.460700 14470 raft_consensus.cc:3037] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:05.461845 14478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.006s sys 0.029s
W20260806 20:36:05.462670 14478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.006s sys 0.029s
W20260806 20:36:05.427479 17625 log.cc:924] Time spent T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 2.055s user 0.000s sys 0.015s
I20260806 20:36:05.395742 17465 raft_consensus.cc:2781] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader election won for term 9
W20260806 20:36:05.428622 14552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 414) took 1574 ms (client timeout 1785 ms). Trace:
W20260806 20:36:05.475452 14552 rpcz_store.cc:269] 0806 20:36:03.854425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.457434 (+603009us) service_pool.cc:227] Handling call
0806 20:36:05.428426 (+970992us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b540018f1d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:05.428564 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562166528}
I20260806 20:36:05.476897 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.477540 14552 raft_consensus.cc:2454] T cf24df4ca64846378f216b30e63e27f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
W20260806 20:36:05.477885 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 331027us
I20260806 20:36:05.478489 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.479157 14552 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 9.
I20260806 20:36:05.479471 17600 consensus_queue.cc:1046] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:36:05.424978 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:05.481532 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:05.481741 17600 raft_consensus.cc:3032] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 20:36:05.428193 17613 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.039s user 0.001s sys 0.035s
I20260806 20:36:05.482124 17645 raft_consensus.cc:482] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:05.482430 17600 raft_consensus.cc:729] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:05.454684 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:05.439400 17537 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.003s sys 0.034s
W20260806 20:36:05.483091 17537 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.914s user 0.003s sys 0.034s
I20260806 20:36:05.482797 17645 raft_consensus.cc:504] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:05.274260 17668 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.483767 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:05.479992 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:05.485061 14552 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:05.485729 17600 consensus_queue.cc:260] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.494448 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:05.459684 17602 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.888s user 0.008s sys 0.021s
I20260806 20:36:05.331027 17542 raft_consensus.cc:504] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.474561 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:05.522682 14552 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
W20260806 20:36:05.482575 17613 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.093s user 0.001s sys 0.037s
I20260806 20:36:05.457660 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:05.488281 17626 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 2.141s user 0.001s sys 0.021s
W20260806 20:36:05.415962 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 1.00429e+06us
I20260806 20:36:05.504748 15151 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:36:05.534602 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:05.431829 14471 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.026s sys 0.027s
W20260806 20:36:05.535352 14471 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.970s user 0.027s sys 0.027s
W20260806 20:36:05.460552 14630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.887s user 0.021s sys 0.014s
I20260806 20:36:05.382579 17491 raft_consensus.cc:686] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [term 8 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:05.459429 17667 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:05.448612 16844 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:05.531045 14625 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.294s user 0.019s sys 0.068s
I20260806 20:36:05.531769 14475 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 8: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:36:05.476492 17235 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.799s user 0.014s sys 0.022s
I20260806 20:36:05.565498 14632 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
W20260806 20:36:05.536006 14630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s user 0.022s sys 0.014s
I20260806 20:36:05.536810 14470 raft_consensus.cc:2454] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:36:05.547189 14596 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.475617 17458 leader_election.cc:288] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:05.451859 17376 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:05.555352 14555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36504 (request call id 423) took 1665 ms (client timeout 1731 ms). Trace:
W20260806 20:36:05.582288 14555 rpcz_store.cc:269] 0806 20:36:03.889790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.455414 (+565624us) service_pool.cc:227] Handling call
0806 20:36:05.555146 (+1099732us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400355f54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:05.555283 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584554368}
I20260806 20:36:05.556205 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1701) took 4480 ms. Trace:
W20260806 20:36:05.557355 14625 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.406s user 0.020s sys 0.069s
I20260806 20:36:05.563880 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.583633 14548 rpcz_store.cc:276] 0806 20:36:01.075711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.489945 (+2414234us) service_pool.cc:227] Handling call
0806 20:36:03.564126 (+ 74181us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:03.908407 (+344281us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:03.909107 (+ 700us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:03.930347 (+ 21240us) log.cc:842] Serialized 26 byte log entry
0806 20:36:05.026364 (+1096017us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:05.026424 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.026524 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:05.555944 (+529420us) raft_consensus.cc:1671] finished
0806 20:36:05.555972 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.556134 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10496,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":59225,"thread_start_us":1350591,"threads_started":2,"wal-append.queue_time_us":1570302}
I20260806 20:36:05.569500 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1706) took 1049 ms. Trace:
I20260806 20:36:05.586448 14471 rpcz_store.cc:276] 0806 20:36:04.519953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.520113 (+ 160us) service_pool.cc:227] Handling call
0806 20:36:04.565353 (+ 45240us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:05.550665 (+985312us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:36:05.550711 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:05.569155 (+ 18444us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:05.569219 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.569300 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.569434 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":158914,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"thread_start_us":969973,"threads_started":1}
W20260806 20:36:05.571822 14551 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.753s user 0.034s sys 0.039s
I20260806 20:36:05.573144 14629 raft_consensus.cc:1261] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:36:05.590009 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:05.460253 17459 consensus_queue.cc:1046] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260806 20:36:05.571511 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:36:05.594404 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 47320.8us
I20260806 20:36:05.528815 17465 raft_consensus.cc:686] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:05.581357 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:05.596650 14470 raft_consensus.cc:2396] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 20:36:05.590528 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1697) took 4689 ms. Trace:
I20260806 20:36:05.549660 17554 raft_consensus.cc:2781] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:05.591466 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:05.614316 14555 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:05.407933 17332 raft_consensus.cc:504] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.593691 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 461) took 1115 ms. Trace:
I20260806 20:36:05.595314 14596 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:05.591234 14551 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.034s sys 0.039s
I20260806 20:36:05.594759 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 454) took 1674 ms. Trace:
I20260806 20:36:05.597101 14553 rpcz_store.cc:276] 0806 20:36:00.901013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.444135 (+2543122us) service_pool.cc:227] Handling call
0806 20:36:03.457231 (+ 13096us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:03.457691 (+ 460us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:03.960116 (+502425us) log.cc:842] Serialized 14 byte log entry
0806 20:36:05.378833 (+1418717us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:05.379296 (+ 463us) log.cc:842] Serialized 26 byte log entry
0806 20:36:05.384101 (+ 4805us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:05.384142 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.384216 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:05.590313 (+206097us) raft_consensus.cc:1671] finished
0806 20:36:05.590332 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.590469 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4559,"prepare.queue_time_us":776031,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":366326,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":63848,"thread_start_us":1618022,"threads_started":3,"wal-append.queue_time_us":943891}
I20260806 20:36:05.615072 14630 rpcz_store.cc:276] 0806 20:36:04.478339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.478642 (+ 303us) service_pool.cc:227] Handling call
0806 20:36:04.479015 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:04.489373 (+ 10358us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:36:04.539774 (+ 50401us) log.cc:842] Serialized 14 byte log entry
0806 20:36:05.577298 (+1037524us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:05.577366 (+ 68us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:05.593388 (+ 16022us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.593486 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.593626 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19599,"prepare.queue_time_us":84495,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":11180,"raft.queue_time_us":31200,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":42515,"thread_start_us":993479,"threads_started":1}
I20260806 20:36:05.616397 14625 rpcz_store.cc:276] 0806 20:36:03.920551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.076062 (+155511us) service_pool.cc:227] Handling call
0806 20:36:04.076543 (+ 481us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:04.076842 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 20:36:04.151352 (+ 74510us) log.cc:842] Serialized 14 byte log entry
0806 20:36:05.594454 (+1443102us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:05.594504 (+ 50us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:36:05.594564 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.594621 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.594713 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10375,"prepare.queue_time_us":115613,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":24544,"thread_start_us":1432308,"threads_started":1}
I20260806 20:36:05.611115 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.619087 14632 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:05.619350 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.493248 17352 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.622390 14596 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:05.615520 17376 raft_consensus.cc:2781] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:05.627499 17376 raft_consensus.cc:482] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:05.627904 17376 raft_consensus.cc:3037] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:05.628276 14632 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:05.633208 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.634073 14553 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:05.635046 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.635713 14553 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:05.527311 17462 consensus_queue.cc:237] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.640930 17465 consensus_queue.cc:237] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:05.644672 14518 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:05.648811 17682 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260806 20:36:05.648693 17684 raft_consensus.cc:2726] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:36:05.650831 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1620) took 3536 ms. Trace:
I20260806 20:36:05.657006 14633 rpcz_store.cc:276] 0806 20:36:02.114540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.236758 (+122218us) service_pool.cc:227] Handling call
0806 20:36:02.237154 (+ 396us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.237501 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.237531 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:02.757417 (+519886us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b2c000b26a8. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:36:02.757920 (+ 503us) log.cc:842] Serialized 26 byte log entry
0806 20:36:03.060974 (+303054us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:03.061021 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:03.061095 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:05.426917 (+2365822us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:05.650632 (+223715us) raft_consensus.cc:1671] finished
0806 20:36:05.650651 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.650774 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105119,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8701,"spinlock_wait_cycles":29265024,"thread_start_us":302625,"threads_started":1,"wal-append.queue_time_us":444428}
I20260806 20:36:05.648334 14518 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.651506 14596 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.655272 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:05.652096 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.662472 14518 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.666898 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:05.667666 14548 raft_consensus.cc:3037] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:05.670400 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 516) took 1851 ms. Trace:
I20260806 20:36:05.670713 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.595999 17661 raft_consensus.cc:2726] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:36:05.670863 15185 rpcz_store.cc:276] 0806 20:36:03.818759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.213491 (+394732us) service_pool.cc:227] Handling call
0806 20:36:04.398775 (+185284us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:04.458588 (+ 59813us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.670211 (+1211623us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b540061ce50. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:05.670346 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":425083776}
I20260806 20:36:05.683621 14548 raft_consensus.cc:2454] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:36:05.670420 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.684566 14553 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6.
I20260806 20:36:05.684612 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.670130 17680 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:36:05.685966 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:05.686614 15151 leader_election.cc:302] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:36:05.687026 14548 raft_consensus.cc:2396] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:36:05.687192 14596 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.687541 15151 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.651327 17688 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:05.692914 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:05.682430 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.695700 14555 raft_consensus.cc:2454] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:05.696007 14552 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 4.
I20260806 20:36:05.688982 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.696655 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d2a67d79c54f588be9e64dce5b2202" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.696904 14596 leader_election.cc:302] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.697865 14555 raft_consensus.cc:2396] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:36:05.698333 15151 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
W20260806 20:36:05.695410 15181 consensus_meta.cc:287] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.796s user 0.012s sys 0.000s
I20260806 20:36:05.660549 17554 raft_consensus.cc:686] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:05.740033 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:05.663828 17580 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:05.541384 17529 consensus_queue.cc:237] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.536787 17457 consensus_queue.cc:237] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.694953 17668 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:05.740895 17491 consensus_queue.cc:237] T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:05.748876 17645 leader_election.cc:288] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:05.800998 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 1.12643e+06us
W20260806 20:36:05.801488 14554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.093s user 0.009s sys 0.072s
W20260806 20:36:05.801748 14554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.100s user 0.009s sys 0.072s
I20260806 20:36:05.802807 14595 leader_election.cc:302] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.749594 17684 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:05.681819 17600 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:05.773331 15181 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:36:05.775527 14624 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:05.692834 17602 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:36:05.693775 17505 consensus_meta.cc:287] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.030s user 0.012s sys 0.000s
W20260806 20:36:05.781241 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 118880us
I20260806 20:36:05.781744 14552 raft_consensus.cc:1226] T 53fd310e10544cb0af683409f6d8e454 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: [7.3-7.3]
I20260806 20:36:05.735730 17577 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:36:05.697458 16844 raft_consensus.cc:2781] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:05.736508 17614 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:05.795992 14444 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:05.797696 14470 raft_consensus.cc:1261] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 20:36:05.801235 14547 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.949s user 0.024s sys 0.039s
I20260806 20:36:05.749094 17680 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:05.805132 14595 leader_election.cc:398] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 3. Current term is 4.
I20260806 20:36:05.831830 14595 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:05.817229 17480 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.017s sys 0.034s
I20260806 20:36:05.751398 17599 raft_consensus.cc:2781] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:05.743574 17688 raft_consensus.cc:686] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:05.878227 17688 consensus_queue.cc:237] T eb32d3aa416443529139823cf8e2d00b P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:05.804306 14441 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.001s sys 0.040s
W20260806 20:36:05.880654 14441 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.704s user 0.002s sys 0.040s
W20260806 20:36:05.881362 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 714785us
I20260806 20:36:05.882164 14441 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.805716 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.885069 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7667fe05a410bbd313d3d601d18b2" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:05.885773 14548 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:05.827082 14552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1703) took 3790 ms. Trace:
I20260806 20:36:05.888537 14552 rpcz_store.cc:276] 0806 20:36:02.036929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.739120 (+3702191us) service_pool.cc:227] Handling call
0806 20:36:05.781462 (+ 42342us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:05.826733 (+ 45271us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.827012 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:05.889856 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:05.890550 14552 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 6
W20260806 20:36:05.834290 14547 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s user 0.025s sys 0.039s
I20260806 20:36:05.893759 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1694) took 5094 ms. Trace:
I20260806 20:36:05.894210 14547 rpcz_store.cc:276] 0806 20:36:00.799618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.376758 (+2577140us) service_pool.cc:227] Handling call
0806 20:36:03.377728 (+ 970us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:03.382028 (+ 4300us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:36:04.851594 (+1469566us) log.cc:842] Serialized 14 byte log entry
0806 20:36:05.893278 (+1041684us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:05.893354 (+ 76us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:05.893421 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.893527 (+ 106us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.893694 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":851471,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"thread_start_us":2476855,"threads_started":2}
I20260806 20:36:05.896149 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:05.919672 14547 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:05.833456 14628 raft_consensus.cc:2396] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 20:36:05.852300 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:05.807914 17580 raft_consensus.cc:2741] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260806 20:36:05.830478 17614 raft_consensus.cc:482] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:05.826467 14518 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:05.830205 17559 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.028s sys 0.024s
I20260806 20:36:05.830016 17577 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:05.855304 16844 raft_consensus.cc:686] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:05.803959 17455 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:05.820838 17602 raft_consensus.cc:482] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:05.867776 14442 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.810119 17690 consensus_queue.cc:1046] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.134s
I20260806 20:36:05.806449 17537 consensus_queue.cc:1035] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
W20260806 20:36:05.820976 17669 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.023s sys 0.029s
W20260806 20:36:05.857884 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 61981.8us
W20260806 20:36:05.912819 17393 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.009s sys 0.051s
W20260806 20:36:05.976732 17403 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.013s sys 0.054s
W20260806 20:36:06.073493 17403 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s user 0.014s sys 0.054s
I20260806 20:36:06.075217 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1690) took 5406 ms. Trace:
I20260806 20:36:05.939656 14441 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:06.077226 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 137756us
W20260806 20:36:06.078792 14476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.013s sys 0.052s
W20260806 20:36:06.079188 14476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.013s sys 0.052s
I20260806 20:36:06.081418 17403 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:06.081228 17661 consensus_queue.cc:1035] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:36:06.081991 17403 raft_consensus.cc:482] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:36:05.934618 17559 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s user 0.028s sys 0.024s
I20260806 20:36:06.082494 17403 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:36:05.943148 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 116768us
I20260806 20:36:06.086593 14518 leader_election.cc:398] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7.
I20260806 20:36:06.087626 14518 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.090281 17455 raft_consensus.cc:3032] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:36:06.090807 17455 raft_consensus.cc:729] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.097923 17668 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:36:06.098459 17689 log.cc:924] Time spent T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.200s user 0.002s sys 0.000s
I20260806 20:36:06.098531 17668 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:06.099264 17668 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:06.106170 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1792) took 5695 ms. Trace:
W20260806 20:36:06.108175 17698 log.cc:924] Time spent T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.178s user 0.000s sys 0.002s
I20260806 20:36:06.112500 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:36:05.947809 15184 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.025s sys 0.032s
W20260806 20:36:06.114293 15184 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.738s user 0.025s sys 0.032s
I20260806 20:36:05.869951 17111 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:05.869725 17691 raft_consensus.cc:482] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:05.867264 17663 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:05.871553 17542 leader_election.cc:288] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:05.869470 17680 raft_consensus.cc:686] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:05.961735 17602 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:36:05.962325 17694 log.cc:924] Time spent T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260806 20:36:05.947999 17566 raft_consensus.cc:2726] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:05.810848 17684 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.224138 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.224321 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b370f85a8b944198972759bc1eb8061" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.225251 15180 raft_consensus.cc:2379] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 8: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:06.225690 17684 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:05.962422 16844 consensus_queue.cc:237] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:05.963305 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 521) took 2143 ms. Trace:
I20260806 20:36:06.226291 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:06.226456 15186 rpcz_store.cc:276] 0806 20:36:03.820030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.368487 (+548457us) service_pool.cc:227] Handling call
0806 20:36:04.826712 (+458225us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:04.827647 (+ 935us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.963081 (+1135434us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b540037f0d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:05.963244 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1052910336}
I20260806 20:36:06.226992 15180 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:06.227722 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:36:06.229051 17681 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
W20260806 20:36:05.964653 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 96997us
W20260806 20:36:06.229844 14628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 456) took 2300 ms (client timeout 2675 ms). Trace:
I20260806 20:36:06.229983 14442 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.231283 17683 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:36:06.231395 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.231707 14442 leader_election.cc:302] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:05.964741 17235 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:36:06.230304 14628 rpcz_store.cc:269] 0806 20:36:03.921083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.962316 (+ 41233us) service_pool.cc:227] Handling call
0806 20:36:06.221168 (+2258852us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54003bf454. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:06.221305 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4242730368}
I20260806 20:36:05.976943 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1679) took 3570 ms. Trace:
I20260806 20:36:06.232633 17235 raft_consensus.cc:482] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:06.232743 14477 rpcz_store.cc:276] 0806 20:36:02.406919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.411173 (+ 4254us) service_pool.cc:227] Handling call
0806 20:36:02.426723 (+ 15550us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:04.262587 (+1835864us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.976654 (+1714067us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54002323d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:05.976867 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4220156928}
W20260806 20:36:05.865880 17695 log.cc:924] Time spent T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260806 20:36:06.233119 17680 consensus_queue.cc:237] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:05.935820 17614 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:06.236754 14473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1701) took 1723 ms. Trace:
I20260806 20:36:06.237102 14473 rpcz_store.cc:276] 0806 20:36:04.512883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.513121 (+ 238us) service_pool.cc:227] Handling call
0806 20:36:04.513540 (+ 419us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:04.513947 (+ 407us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:04.513984 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:04.594706 (+ 80722us) log.cc:842] Serialized 26 byte log entry
0806 20:36:05.410659 (+815953us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:05.410700 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.410769 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:06.236542 (+825773us) raft_consensus.cc:1671] finished
0806 20:36:06.236565 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:06.236689 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":210482,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"thread_start_us":791715,"threads_started":1,"wal-append.queue_time_us":1001537}
I20260806 20:36:05.947499 17577 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:36:05.982857 17669 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.023s sys 0.029s
I20260806 20:36:06.000409 14548 raft_consensus.cc:2454] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:06.240798 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.241367 14548 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:36:05.829409 15181 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59394 (request call id 523) took 1174 ms (client timeout 1561 ms). Trace:
W20260806 20:36:06.244585 15181 rpcz_store.cc:269] 0806 20:36:04.655206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.663693 (+ 8487us) service_pool.cc:227] Handling call
0806 20:36:05.827810 (+1164117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.246031 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:05.999958 14552 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
W20260806 20:36:05.980715 17393 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s user 0.009s sys 0.052s
W20260806 20:36:05.985800 17646 consensus_meta.cc:287] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.828s user 0.013s sys 0.000s
I20260806 20:36:06.247890 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.248139 17646 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.248616 14552 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:06.249353 17459 raft_consensus.cc:2726] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:36:05.955284 17332 leader_election.cc:288] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:06.250113 17646 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
W20260806 20:36:05.994501 17480 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.017s sys 0.034s
I20260806 20:36:06.133224 17691 raft_consensus.cc:504] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:05.948127 17696 consensus_queue.cc:1046] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260806 20:36:06.233137 17235 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:06.253037 17696 raft_consensus.cc:3032] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:36:06.253455 17696 raft_consensus.cc:729] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.253691 17691 leader_election.cc:288] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:05.988103 17352 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:06.056600 15150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.030s sys 0.052s
W20260806 20:36:06.254402 15150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.805s user 0.030s sys 0.053s
I20260806 20:36:06.254303 17696 consensus_queue.cc:260] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.255637 17696 raft_consensus.cc:3037] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:06.071225 17155 raft_consensus.cc:3032] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:06.256623 17155 raft_consensus.cc:729] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.247897 17393 raft_consensus.cc:482] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:06.257305 17393 raft_consensus.cc:504] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.257515 17155 consensus_queue.cc:260] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.258852 17155 raft_consensus.cc:3037] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:36:06.259032 17667 consensus_meta.cc:287] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.687s user 0.008s sys 0.000s
I20260806 20:36:06.259250 17393 leader_election.cc:288] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:06.259526 17667 raft_consensus.cc:504] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.072718 17258 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:06.261148 14548 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:06.121989 15184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59394 (request call id 520) took 2302 ms. Trace:
I20260806 20:36:06.001202 17690 consensus_queue.cc:1046] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260806 20:36:06.075735 14546 rpcz_store.cc:276] 0806 20:36:00.669157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.403324 (+1734167us) service_pool.cc:227] Handling call
0806 20:36:02.403561 (+ 237us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:04.821471 (+2417910us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.074902 (+1253431us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b540018e2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:06.075141 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5369040128}
I20260806 20:36:06.169780 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:06.180243 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:06.262351 17690 raft_consensus.cc:3032] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:36:06.262485 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.264433 17690 raft_consensus.cc:729] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:36:06.265188 17536 consensus_meta.cc:287] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 1.535s user 0.008s sys 0.006s
I20260806 20:36:06.261713 15184 rpcz_store.cc:276] 0806 20:36:03.819752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.221994 (+402242us) service_pool.cc:227] Handling call
0806 20:36:04.222506 (+ 512us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:04.223909 (+ 1403us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:04.223954 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:04.254121 (+ 30167us) log.cc:842] Serialized 26 byte log entry
0806 20:36:05.348447 (+1094326us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:06.114693 (+766246us) log.cc:842] Serialized 14 byte log entry
0806 20:36:06.121564 (+ 6871us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.121678 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:06.121741 (+ 63us) raft_consensus.cc:1671] finished
0806 20:36:06.121757 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:06.121920 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27388,"prepare.queue_time_us":55177,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":137875,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":198,"thread_start_us":1648613,"threads_started":2,"wal-append.queue_time_us":1270438}
I20260806 20:36:06.091810 17455 consensus_queue.cc:260] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.265974 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.265621 17536 raft_consensus.cc:504] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.265563 17690 consensus_queue.cc:260] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.266584 15184 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:06.266660 17455 raft_consensus.cc:3037] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:06.267129 17690 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:06.267495 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.106741 14550 rpcz_store.cc:276] 0806 20:36:00.410414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:02.142784 (+1732370us) service_pool.cc:227] Handling call
0806 20:36:02.937140 (+794356us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:02.938734 (+ 1594us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:02.938778 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:03.129167 (+190389us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b2c00000238. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b1ed6e11 00007fb9b1ee6442 00007fb9b352ec11 00007fb9b34f358a 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772
0806 20:36:03.129549 (+ 382us) log.cc:842] Serialized 26 byte log entry
0806 20:36:04.906230 (+1776681us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:04.906284 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:04.906382 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:05.258275 (+351893us) raft_consensus.cc:1671] finished
0806 20:36:05.258295 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:06.105952 (+847657us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54001faed0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:06.106096 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":758,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":16812,"spinlock_wait_cycles":1396223232,"thread_start_us":1776240,"threads_started":1,"wal-append.queue_time_us":2113437}
I20260806 20:36:06.269716 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1804) took 5479 ms. Trace:
I20260806 20:36:06.128746 17645 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:06.232420 15183 raft_consensus.cc:2379] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:06.271180 17645 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.272409 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:36:06.230795 17649 consensus_meta.cc:287] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.112s user 0.012s sys 0.000s
W20260806 20:36:06.273820 17302 consensus_meta.cc:287] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.875s user 0.010s sys 0.000s
I20260806 20:36:06.273439 17649 raft_consensus.cc:504] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.274221 17302 raft_consensus.cc:2726] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:36:06.214162 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.275795 17649 leader_election.cc:288] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:06.277027 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.277668 15184 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:06.209371 17703 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:06.278502 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:06.278566 17703 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.276296 17614 raft_consensus.cc:504] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.118307 17613 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:06.219996 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.291235 15179 raft_consensus.cc:2454] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:36:06.119036 17486 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:06.117677 17111 raft_consensus.cc:482] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:36:06.222826 17692 log.cc:924] Time spent T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.428s user 0.001s sys 0.000s
I20260806 20:36:06.228950 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:05.963608 17530 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:06.231731 17683 raft_consensus.cc:482] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:05.976601 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 411) took 4758 ms. Trace:
W20260806 20:36:06.291954 17638 consensus_meta.cc:287] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.476s user 0.011s sys 0.008s
I20260806 20:36:05.870625 17554 consensus_queue.cc:237] T 6d7dc7fd91be46ecb624803a9b029e5b P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.260741 17258 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.270705 14550 raft_consensus.cc:1226] T f5277702c39f453cb1063cf506183484 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:06.273626 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:06.272352 14552 raft_consensus.cc:2454] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:06.363031 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.363691 14552 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:06.282052 14596 leader_election.cc:302] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:06.214486 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.212627 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bd288a57145e8b254611b1dd82586" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.270094 14554 rpcz_store.cc:276] 0806 20:36:00.790353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.250986 (+2460633us) service_pool.cc:227] Handling call
0806 20:36:04.005950 (+754964us) spinlock_profiling.cc:243] Waited 6.51 ms on lock 0x7b58002b4798. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84492b0 00007fb9b84347f9 00007fb9b84342a2 00007fb9b1b1a684
0806 20:36:04.006250 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:04.476974 (+470724us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:04.477029 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:04.477557 (+ 528us) log.cc:842] Serialized 26 byte log entry
0806 20:36:05.802050 (+1324493us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:36:05.802089 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.802155 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:06.269431 (+467276us) raft_consensus.cc:1671] finished
0806 20:36:06.269449 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:06.269658 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":496402,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":68738,"spinlock_wait_cycles":14979328,"thread_start_us":1568012,"threads_started":2,"wal-append.queue_time_us":1660219}
I20260806 20:36:06.366674 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:06.211236 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:06.288394 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 56751.6us
I20260806 20:36:06.291980 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.225054 15182 raft_consensus.cc:2454] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 8.
I20260806 20:36:06.308633 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:06.292729 14518 connection.cc:232] RPC call timeout handler was delayed by -5.2796451229951344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
I20260806 20:36:05.986877 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:06.369668 14547 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:06.362409 14628 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:36:06.316517 15185 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:36:06.318416 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:06.371599 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 89636.2us
I20260806 20:36:06.328349 15179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:36:06.371299 14518 connection.cc:232] RPC call timeout handler was delayed by -4.77114512299886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
W20260806 20:36:06.372989 14518 connection.cc:232] RPC call timeout handler was delayed by -4.5128451230120845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23610000000000003s for a response
I20260806 20:36:06.373206 14521 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:06.373975 14518 connection.cc:232] RPC call timeout handler was delayed by -2.6930451230145991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4383s for a response
I20260806 20:36:06.371039 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1706) took 4333 ms. Trace:
I20260806 20:36:06.374717 14442 leader_election.cc:302] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.374809 14550 rpcz_store.cc:276] 0806 20:36:02.038000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.270205 (+4232205us) service_pool.cc:227] Handling call
0806 20:36:06.270537 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.370781 (+100244us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.370970 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:06.296401 17682 log.cc:924] Time spent T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
W20260806 20:36:06.374363 14518 connection.cc:232] RPC call timeout handler was delayed by -2.6615451230027247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4418s for a response
I20260806 20:36:06.267769 17536 leader_election.cc:288] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:06.387478 14626 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.013s sys 0.011s
I20260806 20:36:06.377936 14547 raft_consensus.cc:1261] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:36:06.370697 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.389590 14442 leader_election.cc:302] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:06.314316 14551 rpcz_store.cc:276] 0806 20:36:01.218661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:03.685259 (+2466598us) service_pool.cc:227] Handling call
0806 20:36:03.685683 (+ 424us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:36:04.813757 (+1128074us) spinlock_profiling.cc:243] Waited 6.15 ms on lock 0x7b180047ef60. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b318282c 00007fb9b05ad9dc 00007fb9b93e4a39 00007fb9b05ca8bb 00007fb9b05c8cbf 00007fb9b05c8155 00007fb9b34eb3d0 00007fb9b34ebf6b 00007fb9b34f678b
0806 20:36:04.813903 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:36:04.818175 (+ 4272us) log.cc:842] Serialized 14 byte log entry
0806 20:36:05.623851 (+805676us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:36:05.624515 (+ 664us) log.cc:842] Serialized 52 byte log entry
0806 20:36:05.624671 (+ 156us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:05.624906 (+ 235us) log.cc:842] Serialized 14 byte log entry
0806 20:36:05.629216 (+ 4310us) log.cc:842] Serialized 14 byte log entry
0806 20:36:05.629885 (+ 669us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.630039 (+ 154us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:05.801439 (+171400us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b180047ef60. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b946a8b1 00007fb9b9477661 00007fb9b05cb764 00007fb9b05ceb11 00007fb9b05ceac9 00007fb9b05cea59 00007fb9b05cea21 00007fb9b05cdd1d 00007fb9b982f204 00007fb9b982f039 00007fb9b05addc6
0806 20:36:05.976311 (+174872us) raft_consensus.cc:1671] finished
0806 20:36:05.976331 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:05.976533 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15185,"prepare.queue_time_us":618335,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":107,"raft.queue_time_us":291891,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":9701,"spinlock_wait_cycles":113840768,"thread_start_us":1075963,"threads_started":2,"wal-append.queue_time_us":1154941}
I20260806 20:36:06.381548 14554 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:06.392709 15183 raft_consensus.cc:2396] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 20:36:06.139840 17663 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.381705 14518 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.399056 14518 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:06.315084 17708 log.cc:924] Time spent T 00000000000000000000000000000000 P 5c7ed2f344954f8c90dbc753df8f30f7: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260806 20:36:06.406419 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.409013 14550 raft_consensus.cc:1226] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:06.409888 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1712) took 4236 ms. Trace:
I20260806 20:36:06.410347 14550 rpcz_store.cc:276] 0806 20:36:02.173513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.375976 (+4202463us) service_pool.cc:227] Handling call
0806 20:36:06.408743 (+ 32767us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.409563 (+ 820us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.409818 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.383940 14470 raft_consensus.cc:2379] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 5: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:36:06.290602 17566 raft_consensus.cc:482] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:06.373219 14469 raft_consensus.cc:2354] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
I20260806 20:36:06.297197 17664 raft_consensus.cc:482] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:06.370040 15181 raft_consensus.cc:2379] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:36:06.378971 14473 raft_consensus.cc:1226] T 58bf5720d4f5403b877975863b1cd46c P 92828dd6dbe74f29a04d4e702b502621 [term 8 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:06.389489 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1711) took 4233 ms. Trace:
I20260806 20:36:06.431448 14547 rpcz_store.cc:276] 0806 20:36:02.155654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.370357 (+4214703us) service_pool.cc:227] Handling call
0806 20:36:06.377629 (+ 7272us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.389100 (+ 11471us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.389416 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.308768 17111 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:06.418987 17459 raft_consensus.cc:2781] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:06.400157 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1713) took 4195 ms. Trace:
I20260806 20:36:06.450917 14554 rpcz_store.cc:276] 0806 20:36:02.204476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.381010 (+4176534us) service_pool.cc:227] Handling call
0806 20:36:06.381351 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.399834 (+ 18483us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.400089 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.452315 14554 raft_consensus.cc:1261] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 20:36:06.401727 14442 leader_election.cc:302] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:06.453717 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 52083.8us
W20260806 20:36:06.454087 14625 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.008s sys 0.016s
W20260806 20:36:06.454572 14625 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.701s user 0.008s sys 0.016s
I20260806 20:36:06.454471 14442 leader_election.cc:302] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:06.455144 15152 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.002s sys 0.022s
W20260806 20:36:06.455536 15152 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.708s user 0.002s sys 0.022s
W20260806 20:36:06.456017 14630 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.024s sys 0.059s
I20260806 20:36:06.415292 14518 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
W20260806 20:36:06.425442 14631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.037s sys 0.084s
W20260806 20:36:06.457448 14631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s user 0.037s sys 0.084s
I20260806 20:36:06.386142 17332 raft_consensus.cc:2781] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:06.399278 17486 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:06.406345 17684 raft_consensus.cc:2781] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:06.439179 14550 raft_consensus.cc:1226] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:06.399140 17458 raft_consensus.cc:2781] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:06.439540 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ce96f4f7f49cfa853221635714edc" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.482583 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 412) took 4036 ms. Trace:
I20260806 20:36:06.483153 14550 rpcz_store.cc:276] 0806 20:36:02.446482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.411442 (+3964960us) service_pool.cc:227] Handling call
0806 20:36:06.438987 (+ 27545us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.482273 (+ 43286us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.482522 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.317889 17683 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:36:06.484972 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.458941 17580 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:36:06.485761 14550 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:06.487109 17458 raft_consensus.cc:686] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.488124 17458 consensus_queue.cc:237] T 91a7667fe05a410bbd313d3d601d18b2 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.488809 14474 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:36:06.447327 14547 raft_consensus.cc:1261] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:36:06.489977 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1809) took 4000 ms. Trace:
I20260806 20:36:06.490351 14547 rpcz_store.cc:276] 0806 20:36:02.489861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.432618 (+3942757us) service_pool.cc:227] Handling call
0806 20:36:06.446953 (+ 14335us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.489687 (+ 42734us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.489917 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.492058 17393 consensus_queue.cc:1035] T 08d2a67d79c54f588be9e64dce5b2202 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 20:36:06.492816 17701 log.cc:924] Time spent T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
W20260806 20:36:06.493752 17705 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.176s user 0.000s sys 0.000s
I20260806 20:36:06.493958 14550 raft_consensus.cc:2454] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:36:06.447932 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.455812 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.494983 17602 raft_consensus.cc:504] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.463048 17459 raft_consensus.cc:686] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.497537 17459 consensus_queue.cc:237] T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.418987 17536 consensus_queue.cc:1035] T 07ca0af1acba4a7ea01bf15f5f887457 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260806 20:36:06.433775 17664 raft_consensus.cc:504] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.503245 17658 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:36:06.503666 17658 raft_consensus.cc:482] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:06.503988 17658 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:06.465651 17352 raft_consensus.cc:2781] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:36:06.418251 17566 raft_consensus.cc:504] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.505585 17663 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:06.506368 17663 consensus_queue.cc:1046] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:36:06.465587 17486 raft_consensus.cc:482] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:06.507516 17486 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:06.507735 17663 consensus_queue.cc:1035] T 56e9ab616aca4e6bb97d457694148205 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:36:06.415562 17668 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:06.459550 17462 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.814s user 0.005s sys 0.020s
I20260806 20:36:06.317821 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.469023 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1811) took 3962 ms. Trace:
I20260806 20:36:06.544440 14554 rpcz_store.cc:276] 0806 20:36:02.490478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.451709 (+3961231us) service_pool.cc:227] Handling call
0806 20:36:06.452008 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.452681 (+ 673us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.452850 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.476207 14475 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 4.
I20260806 20:36:06.482664 17614 leader_election.cc:288] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:06.478794 17684 raft_consensus.cc:482] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:06.358831 17530 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.506724 17566 leader_election.cc:288] T 53bd3d0855da421aac4540ae1bfbb8f3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:06.508662 17559 consensus_queue.cc:1046] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:36:06.509415 17537 raft_consensus.cc:3032] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Stepping down as leader of term 7
W20260806 20:36:06.463406 17600 consensus_meta.cc:287] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.643s user 0.009s sys 0.001s
I20260806 20:36:06.512615 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:06.514362 17683 raft_consensus.cc:504] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.514744 17258 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:06.515452 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:06.463027 17555 raft_consensus.cc:2726] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:36:06.518327 17403 raft_consensus.cc:504] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.453907 17703 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:06.520423 17376 consensus_meta.cc:287] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.892s user 0.012s sys 0.000s
I20260806 20:36:06.521018 17658 raft_consensus.cc:504] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.522069 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.523290 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:06.598161 14477 raft_consensus.cc:2379] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 10: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:36:06.562602 17566 raft_consensus.cc:2726] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:06.519816 17577 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:06.437325 14479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.041s sys 0.069s
W20260806 20:36:06.606304 14479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s user 0.042s sys 0.070s
I20260806 20:36:06.528755 15187 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 5.
I20260806 20:36:06.453935 14552 raft_consensus.cc:2454] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:06.608562 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:06.459328 14630 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.686s user 0.024s sys 0.059s
W20260806 20:36:06.460445 14626 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s user 0.014s sys 0.011s
W20260806 20:36:06.538579 14596 leader_election.cc:334] T d8a0e4079ebe47aaa3c275ed7f559ab8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545): Timed out: RequestConsensusVote RPC to 127.12.248.130:39545 timed out after 1.657s (SENT)
I20260806 20:36:06.609778 14552 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
W20260806 20:36:06.612071 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 73531us
I20260806 20:36:06.612722 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.613384 14552 raft_consensus.cc:3037] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:06.616078 14442 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:36:06.613814 14597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.000s sys 0.027s
W20260806 20:36:06.616807 14597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.798s user 0.000s sys 0.027s
W20260806 20:36:06.616878 14632 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.009s sys 0.015s
W20260806 20:36:06.617502 14632 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.813s user 0.009s sys 0.015s
I20260806 20:36:06.504144 17664 leader_election.cc:288] T e391f934d63c4ad788870d7abc39a79c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:06.455943 17465 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.000s sys 0.026s
I20260806 20:36:06.545662 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:06.545298 15151 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:36:06.548228 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:06.537220 17332 raft_consensus.cc:686] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:06.620723 14554 raft_consensus.cc:2454] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:36:06.541702 17667 leader_election.cc:288] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:06.544173 17462 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s user 0.006s sys 0.020s
W20260806 20:36:06.577087 14555 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.783s user 0.012s sys 0.015s
I20260806 20:36:06.550482 17684 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:36:06.569427 17491 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.007s sys 0.017s
I20260806 20:36:06.580519 14547 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:06.594447 14470 raft_consensus.cc:2379] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
W20260806 20:36:06.524027 17255 consensus_meta.cc:287] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 1.108s user 0.008s sys 0.002s
I20260806 20:36:06.562883 17537 raft_consensus.cc:729] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.602530 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289df1801e0045beb52c1aaaf43e73f0" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:06.622946 17663 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:36:06.610880 14550 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
W20260806 20:36:06.618711 15153 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.005s sys 0.019s
I20260806 20:36:06.621497 14629 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:06.540302 17302 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:06.622599 14596 leader_election.cc:302] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:06.623010 15184 raft_consensus.cc:2354] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5.
W20260806 20:36:06.623556 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 78338.6us
I20260806 20:36:06.630110 17332 consensus_queue.cc:237] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.603844 17235 raft_consensus.cc:504] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.603430 17376 raft_consensus.cc:504] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.632551 17258 raft_consensus.cc:2781] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:06.634099 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:06.634150 17577 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:06.637189 17543 leader_election.cc:288] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:06.637446 17403 leader_election.cc:288] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:06.609988 17602 leader_election.cc:288] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:06.640709 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
W20260806 20:36:06.642035 14555 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s user 0.012s sys 0.015s
I20260806 20:36:06.620555 17531 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:06.664938 17668 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:06.666637 17658 leader_election.cc:288] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:06.668560 14595 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.808s user 0.018s sys 0.009s
I20260806 20:36:06.635700 17486 raft_consensus.cc:504] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:06.623047 17465 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.979s user 0.000s sys 0.026s
I20260806 20:36:06.675576 14546 raft_consensus.cc:1226] T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:06.678933 14442 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:06.697540 17663 raft_consensus.cc:482] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:06.740765 17663 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:36:06.698935 15153 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.913s user 0.006s sys 0.019s
W20260806 20:36:06.744956 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 958786us
I20260806 20:36:06.700073 14553 raft_consensus.cc:2396] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260806 20:36:06.701156 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 78621.1us
I20260806 20:36:06.703765 15151 leader_election.cc:302] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:36:06.717150 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.753584 14547 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:06.680963 14441 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.598057 14471 raft_consensus.cc:2362] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 6. Re-sending same reply.
W20260806 20:36:06.624332 17529 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.011s sys 0.015s
W20260806 20:36:06.671715 17491 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s user 0.007s sys 0.017s
I20260806 20:36:06.671797 17605 raft_consensus.cc:3032] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:36:06.635728 17393 consensus_queue.cc:1035] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260806 20:36:06.722244 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1718) took 1677 ms. Trace:
W20260806 20:36:06.758970 15150 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.011s sys 0.010s
W20260806 20:36:06.752187 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 48539.9us
I20260806 20:36:06.723544 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1704) took 4686 ms. Trace:
W20260806 20:36:06.729946 14595 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.018s sys 0.009s
W20260806 20:36:06.760983 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 900270us
I20260806 20:36:06.717319 14628 raft_consensus.cc:3037] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:36:06.759308 15150 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.502s user 0.011s sys 0.010s
W20260806 20:36:06.767674 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 511035us
W20260806 20:36:06.752625 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 73821.5us
W20260806 20:36:06.754865 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 74021.9us
I20260806 20:36:06.726118 14473 raft_consensus.cc:2454] T 289df1801e0045beb52c1aaaf43e73f0 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 8.
I20260806 20:36:06.758535 14550 rpcz_store.cc:276] 0806 20:36:05.044605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.494665 (+1450060us) service_pool.cc:227] Handling call
0806 20:36:06.607392 (+112727us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.698865 (+ 91473us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.721988 (+ 23123us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54003e8ad0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:06.722173 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":258488192}
I20260806 20:36:06.771772 14441 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.772975 17403 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:36:06.773522 17403 raft_consensus.cc:482] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:06.773227 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.773028 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:06.773927 17403 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:06.775131 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.775667 14550 raft_consensus.cc:3037] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:06.775557 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.776778 14473 raft_consensus.cc:2379] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 8: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:36:06.776737 17543 consensus_queue.cc:1046] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260806 20:36:06.681820 17599 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.061s sys 0.101s
W20260806 20:36:06.738965 14624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.011s sys 0.008s
W20260806 20:36:06.780864 14624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.012s sys 0.008s
I20260806 20:36:06.785527 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:06.757520 17529 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s user 0.012s sys 0.015s
I20260806 20:36:06.791093 14628 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:06.734231 17602 raft_consensus.cc:2781] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:06.800592 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c10175264634fcd88e06ab41ee2a25c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:36:06.756913 16844 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.014s sys 0.002s
I20260806 20:36:06.811321 14547 raft_consensus.cc:2454] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:06.805744 14628 raft_consensus.cc:3037] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:06.757884 17605 raft_consensus.cc:729] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.724653 17679 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:06.778633 17658 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:36:06.778389 17703 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:36:06.725898 17258 raft_consensus.cc:686] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
W20260806 20:36:06.784793 17599 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.835s user 0.062s sys 0.102s
I20260806 20:36:06.788105 14441 leader_election.cc:302] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:36:06.720921 15180 raft_consensus.cc:2454] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
W20260806 20:36:06.908066 14633 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.016s sys 0.029s
I20260806 20:36:06.818037 14442 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:06.818934 15150 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.814193 14552 raft_consensus.cc:2454] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:06.868441 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:06.791787 17685 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.891s user 0.005s sys 0.024s
W20260806 20:36:06.911263 14633 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.016s sys 0.029s
I20260806 20:36:06.866065 14628 raft_consensus.cc:2454] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:06.917371 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:06.918418 14628 raft_consensus.cc:2379] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:06.919466 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370a97aea0534403aeb11fe85b5a1ca9" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:06.893174 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:06.808588 17352 raft_consensus.cc:686] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:06.716133 17302 raft_consensus.cc:504] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:06.795066 17718 raft_consensus.cc:2781] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:36:06.934571 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 519327us
I20260806 20:36:06.934978 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8933c210fdc40cab4369e6aa56f7020" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:06.913751 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 125709us
I20260806 20:36:06.814841 14550 raft_consensus.cc:2454] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 5.
I20260806 20:36:06.937479 14550 raft_consensus.cc:1226] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:06.870898 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:06.937928 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:06.938361 14547 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Advancing to term 10
W20260806 20:36:06.817113 16844 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s user 0.015s sys 0.003s
W20260806 20:36:06.802991 17688 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.000s sys 0.027s
I20260806 20:36:06.921720 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:06.813143 17726 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:06.813427 17683 leader_election.cc:288] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 10 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:06.956503 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:06.957369 14628 raft_consensus.cc:2379] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 in current term 7: Already voted for candidate df1245ecd2034e11ae6cf688c1c10a4d in this term.
I20260806 20:36:06.724869 17537 consensus_queue.cc:260] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:06.928155 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 109306us
I20260806 20:36:06.961278 15150 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:06.882269 17658 raft_consensus.cc:482] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:36:06.939745 14548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.015s sys 0.014s
W20260806 20:36:06.916642 17685 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.016s user 0.008s sys 0.025s
I20260806 20:36:06.897732 17703 raft_consensus.cc:482] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:06.943109 14441 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:36:06.955451 17688 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.000s sys 0.032s
I20260806 20:36:06.951560 17531 raft_consensus.cc:482] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:36:06.951792 17716 log.cc:924] Time spent T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20260806 20:36:06.952543 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 426) took 1889 ms. Trace:
I20260806 20:36:07.011544 14550 rpcz_store.cc:276] 0806 20:36:05.051163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.936891 (+1885728us) service_pool.cc:227] Handling call
0806 20:36:06.937253 (+ 362us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.939748 (+ 2495us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.939958 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:06.953131 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f645749c8846cf8c2f4dd196506e4d" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:06.920218 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 102273us
I20260806 20:36:06.916270 17543 raft_consensus.cc:2726] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:06.959282 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:06.970794 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:06.906188 17599 raft_consensus.cc:686] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
W20260806 20:36:06.972323 14548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s user 0.016s sys 0.014s
I20260806 20:36:06.952379 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:06.992897 15186 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 20:36:06.927188 17690 raft_consensus.cc:2781] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:36:06.999336 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 56354.1us
I20260806 20:36:06.948949 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1652) took 1203 ms. Trace:
I20260806 20:36:07.027840 14630 rpcz_store.cc:276] 0806 20:36:05.745340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.745477 (+ 137us) service_pool.cc:227] Handling call
0806 20:36:05.745824 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:05.746051 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:36:06.610488 (+864437us) log.cc:842] Serialized 14 byte log entry
0806 20:36:06.948209 (+337721us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:06.948278 (+ 69us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:06.948337 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.948415 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:06.948888 (+ 473us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33426,"prepare.queue_time_us":1005008,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":7301,"raft.queue_time_us":98580,"thread_start_us":1148587,"threads_started":2}
I20260806 20:36:06.947526 17577 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:06.934706 17718 raft_consensus.cc:482] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:07.005757 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc7f43dcc54dc48f61c69d3d4f99a3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:36:06.970896 17457 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.000s sys 0.031s
I20260806 20:36:07.019384 14442 leader_election.cc:302] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:07.022758 14471 raft_consensus.cc:2379] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:36:06.997349 17658 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:07.036501 17543 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:06.998693 17729 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:07.005724 17703 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Advancing to term 9
W20260806 20:36:07.055848 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 286640us
I20260806 20:36:06.996647 17722 raft_consensus.cc:2726] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 20:36:06.999069 17600 consensus_queue.cc:1159] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" responder_term: 8 status { last_received { term: 7 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:36:07.021973 17376 leader_election.cc:288] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:06.960041 17605 consensus_queue.cc:260] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:06.958963 17726 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.058879 14595 leader_election.cc:302] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871
I20260806 20:36:07.011611 17531 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.009805 17459 raft_consensus.cc:2781] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:07.101806 17302 leader_election.cc:288] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.081589 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1816) took 1906 ms. Trace:
I20260806 20:36:07.166333 17537 raft_consensus.cc:3037] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.166375 14550 rpcz_store.cc:276] 0806 20:36:05.175225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.012629 (+1837404us) service_pool.cc:227] Handling call
0806 20:36:07.055273 (+ 42644us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.081196 (+ 25923us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:07.081242 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:07.081313 (+ 71us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:07.081354 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.081419 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.081538 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":839,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":29382}
I20260806 20:36:07.168071 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:07.169314 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:07.169874 14550 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:07.165570 17459 raft_consensus.cc:686] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [term 8 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:07.025570 17486 leader_election.cc:288] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
W20260806 20:36:07.170723 17724 log.cc:924] Time spent T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20260806 20:36:07.059386 15180 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 4.
I20260806 20:36:07.171367 17459 consensus_queue.cc:237] T 7815d5557ccf4e569039960d0a8f24ba P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.172567 17235 leader_election.cc:288] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.172695 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:07.174027 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
W20260806 20:36:07.170389 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 111607us
W20260806 20:36:07.102264 17717 log.cc:924] Time spent T 87adab531acb4032af0c9a18dd9893e0 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.316s user 0.001s sys 0.000s
I20260806 20:36:07.179143 14595 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:07.105049 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55591983ce42868e62df33ee1b56be" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:07.107470 14629 raft_consensus.cc:2454] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 8.
I20260806 20:36:07.175473 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1650) took 1481 ms. Trace:
I20260806 20:36:07.136713 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1639) took 1584 ms. Trace:
I20260806 20:36:07.020605 17458 raft_consensus.cc:482] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:07.180503 14625 rpcz_store.cc:276] 0806 20:36:05.693633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.694510 (+ 877us) service_pool.cc:227] Handling call
0806 20:36:05.694849 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:05.695111 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:06.478539 (+783428us) log.cc:842] Serialized 14 byte log entry
0806 20:36:07.175035 (+696496us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:07.175090 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:07.175149 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.175259 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.175403 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1023406,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":159415,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":9292,"spinlock_wait_cycles":23319808,"thread_start_us":1396536,"threads_started":3,"wal-append.queue_time_us":633291}
I20260806 20:36:07.190196 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
W20260806 20:36:07.125592 17732 log.cc:924] Time spent T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20260806 20:36:07.102288 17704 log.cc:924] Time spent T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.617s user 0.002s sys 0.014s
I20260806 20:36:07.180764 14626 rpcz_store.cc:276] 0806 20:36:05.552422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.552590 (+ 168us) service_pool.cc:227] Handling call
0806 20:36:05.553002 (+ 412us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:05.570998 (+ 17996us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:05.571049 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:05.608778 (+ 37729us) log.cc:842] Serialized 26 byte log entry
0806 20:36:06.611674 (+1002896us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:06.611708 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.611816 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:07.136474 (+524658us) raft_consensus.cc:1671] finished
0806 20:36:07.136501 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.136648 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18564,"raft.queue_time_us":17655,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":130,"thread_start_us":941221,"threads_started":1,"wal-append.queue_time_us":1093235}
W20260806 20:36:07.078967 17699 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 1.599s user 0.003s sys 0.012s
W20260806 20:36:07.124807 17723 log.cc:924] Time spent T 289df1801e0045beb52c1aaaf43e73f0 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.311s user 0.000s sys 0.001s
W20260806 20:36:07.083992 17457 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.000s sys 0.033s
W20260806 20:36:07.087800 17455 consensus_meta.cc:287] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.821s user 0.014s sys 0.000s
I20260806 20:36:07.079020 17690 raft_consensus.cc:686] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:36:07.203681 17735 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260806 20:36:07.204420 17715 log.cc:924] Time spent T d8a0e4079ebe47aaa3c275ed7f559ab8 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
I20260806 20:36:07.191164 17599 consensus_queue.cc:237] T 53fd310e10544cb0af683409f6d8e454 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:07.206571 17713 log.cc:924] Time spent T db55591983ce42868e62df33ee1b56be P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
I20260806 20:36:07.187513 17728 raft_consensus.cc:2781] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:07.242353 17728 raft_consensus.cc:482] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:07.242919 17728 raft_consensus.cc:3037] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.188651 17543 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.245886 17543 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
W20260806 20:36:07.216193 17684 consensus_meta.cc:287] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff: Time spent flushing consensus metadata: real 0.519s user 0.003s sys 0.009s
I20260806 20:36:07.123677 17688 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:07.217885 17663 raft_consensus.cc:504] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:07.192278 17714 log.cc:924] Time spent T bef4a42f075d4ee1ac14ba3159109a18 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.400s user 0.000s sys 0.001s
I20260806 20:36:07.128153 17718 raft_consensus.cc:3037] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.184953 17729 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:07.265637 17744 log.cc:924] Time spent T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.141s user 0.002s sys 0.000s
W20260806 20:36:07.267234 17736 log.cc:924] Time spent T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260806 20:36:07.267493 17703 raft_consensus.cc:504] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:07.259058 17711 log.cc:924] Time spent T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.799s user 0.000s sys 0.003s
I20260806 20:36:07.269768 17703 leader_election.cc:288] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.305780 14469 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:07.308342 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1710) took 2787 ms. Trace:
I20260806 20:36:07.308342 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35910 (request call id 409) took 1774 ms. Trace:
I20260806 20:36:07.308966 14478 rpcz_store.cc:276] 0806 20:36:04.521167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.708441 (+187274us) service_pool.cc:227] Handling call
0806 20:36:04.732778 (+ 24337us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:04.733061 (+ 283us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:04.733093 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:04.737909 (+ 4816us) log.cc:842] Serialized 26 byte log entry
0806 20:36:05.462991 (+725082us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:05.463018 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:05.463065 (+ 47us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:06.861132 (+1398067us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:07.307594 (+446462us) raft_consensus.cc:1671] finished
0806 20:36:07.307635 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.308074 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40488832,"thread_start_us":667227,"threads_started":1,"wal-append.queue_time_us":737427}
I20260806 20:36:07.309513 14479 rpcz_store.cc:276] 0806 20:36:05.533921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.534962 (+ 1041us) service_pool.cc:227] Handling call
0806 20:36:05.579219 (+ 44257us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:05.608930 (+ 29711us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:05.608971 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:05.622687 (+ 13716us) log.cc:842] Serialized 26 byte log entry
0806 20:36:06.606774 (+984087us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:06.606829 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.606932 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:07.101572 (+494640us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:07.307495 (+205923us) raft_consensus.cc:1671] finished
0806 20:36:07.307591 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.308097 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35667,"raft.queue_time_us":36431,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":28089,"spinlock_wait_cycles":24978816,"thread_start_us":904413,"threads_started":1,"wal-append.queue_time_us":1051457}
I20260806 20:36:07.312821 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.313328 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.316619 14476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1705) took 2797 ms. Trace:
I20260806 20:36:07.077785 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.317638 14477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1797) took 1008 ms. Trace:
I20260806 20:36:07.069303 14547 raft_consensus.cc:2454] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 10.
I20260806 20:36:07.324705 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 465) took 1835 ms. Trace:
I20260806 20:36:07.317015 14476 rpcz_store.cc:276] 0806 20:36:04.519518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.519648 (+ 130us) service_pool.cc:227] Handling call
0806 20:36:04.570315 (+ 50667us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:05.405531 (+835216us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:05.405583 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:05.411023 (+ 5440us) log.cc:842] Serialized 26 byte log entry
0806 20:36:06.079589 (+668566us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:06.079643 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.079741 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:07.258313 (+1178572us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b44000232cc. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b04dda95 00007fb9b04de4b2 00007fb9b3186511 00007fb9b3181e1a 00007fb9b3175bee 00007fb9b3174f36 00007fb9b3174b58 00007fb9b317e551 00007fb9b317e509 00007fb9b317e499
0806 20:36:07.315919 (+ 57606us) raft_consensus.cc:1671] finished
0806 20:36:07.315958 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.316420 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":154118,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"spinlock_wait_cycles":81608576,"thread_start_us":1502848,"threads_started":2,"wal-append.queue_time_us":1017515}
I20260806 20:36:07.314865 14474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1794) took 1099 ms. Trace:
I20260806 20:36:07.325222 14631 rpcz_store.cc:276] 0806 20:36:05.488871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.489126 (+ 255us) service_pool.cc:227] Handling call
0806 20:36:05.489455 (+ 329us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:05.489786 (+ 331us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:05.489812 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:05.490205 (+ 393us) log.cc:842] Serialized 26 byte log entry
0806 20:36:06.457902 (+967697us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:06.457966 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.458051 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:07.324064 (+866013us) raft_consensus.cc:1671] finished
0806 20:36:07.324107 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.324469 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":656,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":35212,"thread_start_us":891906,"threads_started":1,"wal-append.queue_time_us":1348958}
I20260806 20:36:07.326475 14474 rpcz_store.cc:276] 0806 20:36:06.215344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.291668 (+ 76324us) service_pool.cc:227] Handling call
0806 20:36:07.314480 (+1022812us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54002aec54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.314696 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":275035520}
I20260806 20:36:07.317056 14475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54502 (request call id 1790) took 1106 ms. Trace:
I20260806 20:36:07.328840 14475 rpcz_store.cc:276] 0806 20:36:06.210717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.210873 (+ 156us) service_pool.cc:227] Handling call
0806 20:36:07.316660 (+1105787us) spinlock_profiling.cc:243] Waited 66.6 ms on lock 0x7b54002af154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.316871 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":153281792}
I20260806 20:36:07.331045 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1709) took 5224 ms. Trace:
W20260806 20:36:07.334849 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 371921us
I20260806 20:36:07.337734 15150 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.191334 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457a5ba876d14681a3fa2338fd9f6c4e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:07.196012 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17eec21790894c559ad9623765626416" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:07.364820 17577 raft_consensus.cc:2726] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:07.365813 17577 raft_consensus.cc:482] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:07.366751 17577 raft_consensus.cc:504] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.318323 14477 rpcz_store.cc:276] 0806 20:36:06.308587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.308704 (+ 117us) service_pool.cc:227] Handling call
0806 20:36:07.316141 (+1007437us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54002175d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.316425 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":477385600}
I20260806 20:36:07.369233 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.370209 14477 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:07.370447 17729 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.371410 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b527f3bc7bb43f88f1b17c445c3c0d0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.321408 14633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38502 (request call id 1923) took 1051 ms. Trace:
I20260806 20:36:07.323280 14626 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:07.083258 15182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:07.199847 17458 raft_consensus.cc:504] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:07.215440 17111 consensus_meta.cc:287] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.756s user 0.006s sys 0.006s
I20260806 20:36:07.382373 17602 raft_consensus.cc:686] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:07.383193 17111 raft_consensus.cc:504] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.209887 17403 raft_consensus.cc:504] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.385028 17458 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.195276 17605 raft_consensus.cc:3037] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:36:07.386731 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 75122.8us
I20260806 20:36:07.387931 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e4ac7b6cd245aead7963fa7cbc5d2e" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.389746 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7dc7fd91be46ecb624803a9b029e5b" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.391170 14470 raft_consensus.cc:2379] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff in current term 6: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:36:07.392386 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:07.320116 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 384623us
I20260806 20:36:07.249887 17684 raft_consensus.cc:504] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.079712 17258 consensus_queue.cc:237] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.372074 14550 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:07.331453 14548 rpcz_store.cc:276] 0806 20:36:02.107128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.278586 (+4171458us) service_pool.cc:227] Handling call
0806 20:36:06.278880 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.279108 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:36:06.279739 (+ 631us) log.cc:842] Serialized 14 byte log entry
0806 20:36:07.329259 (+1049520us) log.cc:842] Serialized 14 byte log entry
0806 20:36:07.330072 (+ 813us) log.cc:842] Serialized 14 byte log entry
0806 20:36:07.330361 (+ 289us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:07.330425 (+ 64us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:07.330514 (+ 89us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.330644 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.330833 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":237397,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":81,"raft.queue_time_us":3962,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":27324,"thread_start_us":1048482,"threads_started":2,"wal-append.queue_time_us":969804}
I20260806 20:36:07.378053 15182 raft_consensus.cc:2354] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8.
I20260806 20:36:07.313663 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fd310e10544cb0af683409f6d8e454" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.111441 17352 consensus_queue.cc:237] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.397779 14625 raft_consensus.cc:1226] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:07.398015 14518 leader_election.cc:302] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.399778 14551 raft_consensus.cc:1226] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:07.401721 17663 leader_election.cc:288] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.402000 17529 raft_consensus.cc:2726] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:36:07.402542 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 383113us
I20260806 20:36:07.402418 17376 consensus_queue.cc:1046] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:36:07.386253 15150 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.407053 17737 raft_consensus.cc:482] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:07.407907 14548 raft_consensus.cc:1226] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:07.372668 14633 rpcz_store.cc:276] 0806 20:36:06.269648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.276597 (+ 6949us) service_pool.cc:227] Handling call
0806 20:36:06.277013 (+ 416us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:06.277284 (+ 271us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:06.277325 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:36:06.277819 (+ 494us) log.cc:842] Serialized 26 byte log entry
0806 20:36:06.922714 (+644895us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:06.922763 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.922844 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:07.320748 (+397904us) raft_consensus.cc:1671] finished
0806 20:36:07.320796 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.321116 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":644583,"threads_started":1,"wal-append.queue_time_us":808405}
I20260806 20:36:07.414875 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1819) took 2236 ms. Trace:
I20260806 20:36:07.415300 14548 rpcz_store.cc:276] 0806 20:36:05.176586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.406976 (+2230390us) service_pool.cc:227] Handling call
0806 20:36:07.407554 (+ 578us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.413064 (+ 5510us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.413204 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.402846 17663 raft_consensus.cc:2781] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:07.420722 17726 leader_election.cc:288] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.423409 17554 consensus_queue.cc:1035] T db55591983ce42868e62df33ee1b56be P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260806 20:36:07.426914 14478 raft_consensus.cc:2396] T 53fd310e10544cb0af683409f6d8e454 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:36:07.430727 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694b5a404a94fbea69b831722e2d3e4" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.432751 14547 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: [3.5-6.7]
I20260806 20:36:07.411800 15178 raft_consensus.cc:3037] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.250327 17688 raft_consensus.cc:504] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.436569 15180 raft_consensus.cc:1226] T 4248ec88bd4b44e990938a74fe3c4fdc P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
W20260806 20:36:07.414640 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 226318us
I20260806 20:36:07.433024 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.259060 17690 consensus_queue.cc:237] T 370a97aea0534403aeb11fe85b5a1ca9 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.403368 17727 raft_consensus.cc:482] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:07.432343 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1705) took 5394 ms. Trace:
I20260806 20:36:07.441793 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.405014 17749 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:07.438603 17663 raft_consensus.cc:686] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:07.446807 14546 rpcz_store.cc:276] 0806 20:36:02.037700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.265502 (+4227802us) service_pool.cc:227] Handling call
0806 20:36:06.265810 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:06.738280 (+472470us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.431825 (+693545us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b540013a1d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:07.432166 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938939264}
I20260806 20:36:07.396373 17602 consensus_queue.cc:237] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.408018 17737 raft_consensus.cc:504] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.466141 17675 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:36:07.467397 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38502 (request call id 1907) took 3193 ms (client timeout 2495 ms). Trace:
I20260806 20:36:07.467145 17675 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:07.467835 14627 rpcz_store.cc:269] 0806 20:36:04.273548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:04.273722 (+ 174us) service_pool.cc:227] Handling call
0806 20:36:07.466955 (+3193233us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b5400313154. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.467210 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5612001664}
I20260806 20:36:07.497521 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40012e131a4bfe9d61a3653f3298c6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:07.469997 14632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1649) took 1789 ms. Trace:
I20260806 20:36:07.471222 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1822) took 2293 ms. Trace:
I20260806 20:36:07.499717 17111 leader_election.cc:288] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.500041 14553 rpcz_store.cc:276] 0806 20:36:05.177476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.685242 (+507766us) service_pool.cc:227] Handling call
0806 20:36:07.470764 (+1785522us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400273dd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.470994 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2305660288}
I20260806 20:36:07.501822 17613 raft_consensus.cc:482] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:07.502362 17613 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.499358 14632 rpcz_store.cc:276] 0806 20:36:05.680766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.680922 (+ 156us) service_pool.cc:227] Handling call
0806 20:36:05.681699 (+ 777us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:05.782812 (+101113us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:36:06.618182 (+835370us) log.cc:842] Serialized 14 byte log entry
0806 20:36:06.950028 (+331846us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:06.950094 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:06.950149 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.469537 (+519388us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400303254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:07.469609 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.469816 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22371,"prepare.queue_time_us":969022,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":152482,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":25501,"spinlock_wait_cycles":231951232,"thread_start_us":1122388,"threads_started":2,"wal-append.queue_time_us":638375}
I20260806 20:36:07.505074 14552 raft_consensus.cc:1261] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260806 20:36:07.505573 14552 pending_rounds.cc:76] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting all ops after (but not including) 3
I20260806 20:36:07.505931 14550 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:07.506848 14550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1823) took 2329 ms. Trace:
I20260806 20:36:07.507243 14550 rpcz_store.cc:276] 0806 20:36:05.177733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.442087 (+2264354us) service_pool.cc:227] Handling call
0806 20:36:07.505613 (+ 63526us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.506542 (+ 929us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.506769 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.471510 14478 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 9.
I20260806 20:36:07.473477 17688 leader_election.cc:288] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:07.474238 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.474795 15181 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 437) took 1271 ms. Trace:
I20260806 20:36:07.475716 14629 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:07.476169 17255 raft_consensus.cc:2726] T 6b370f85a8b944198972759bc1eb8061 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:07.476362 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 435) took 1305 ms. Trace:
I20260806 20:36:07.537706 15183 rpcz_store.cc:276] 0806 20:36:06.170984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.272145 (+101161us) service_pool.cc:227] Handling call
0806 20:36:07.475834 (+1203689us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54006504d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.476151 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":274494336}
I20260806 20:36:07.538152 17531 raft_consensus.cc:504] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.539666 15183 raft_consensus.cc:1226] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [term 7 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:07.541471 17457 raft_consensus.cc:482] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 148bf2a11dba423ab364371f0458e871)
I20260806 20:36:07.478008 15187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60780 (request call id 434) took 1308 ms. Trace:
I20260806 20:36:07.478135 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:07.482203 17675 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:07.474792 17749 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.542549 14624 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:07.542217 17457 raft_consensus.cc:504] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.544091 17749 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:07.484051 17727 raft_consensus.cc:504] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.485085 15151 leader_election.cc:302] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:36:07.485666 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.487497 17684 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.494149 17737 leader_election.cc:288] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.495170 17530 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.452574 17554 raft_consensus.cc:2726] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:07.515528 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568833ac14f74d16a95723bbfa721a96" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.505898 14552 pending_rounds.cc:97] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260806 20:36:07.519212 15181 rpcz_store.cc:276] 0806 20:36:06.203784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.245725 (+ 41941us) service_pool.cc:227] Handling call
0806 20:36:07.474285 (+1228560us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b54004aef54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.474575 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77504896}
I20260806 20:36:07.551033 17580 consensus_queue.cc:1046] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:36:07.552162 17580 raft_consensus.cc:3032] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:36:07.552604 17580 raft_consensus.cc:729] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:07.552968 14633 raft_consensus.cc:1226] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:06.759837 14555 rpcz_store.cc:276] 0806 20:36:02.037250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.769530 (+3732280us) service_pool.cc:227] Handling call
0806 20:36:05.769923 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:05.770180 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 20:36:05.793962 (+ 23782us) log.cc:842] Serialized 14 byte log entry
0806 20:36:06.723109 (+929147us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:06.723188 (+ 79us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:36:06.723257 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:06.723342 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:06.723479 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":31193,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":961,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":4631,"spinlock_wait_cycles":18552064,"thread_start_us":928708,"threads_started":1,"wal-append.queue_time_us":931158}
I20260806 20:36:07.484506 17403 leader_election.cc:288] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.542217 15187 rpcz_store.cc:276] 0806 20:36:06.169302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.169438 (+ 136us) service_pool.cc:227] Handling call
0806 20:36:07.477585 (+1308147us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b54003be2d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.477796 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43523456}
I20260806 20:36:07.512390 15180 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 682) took 1034 ms. Trace:
I20260806 20:36:07.549760 14552 raft_consensus.cc:2866] T 58bf5720d4f5403b877975863b1cd46c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:36:07.550958 17646 consensus_queue.cc:1046] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:36:07.552723 14473 raft_consensus.cc:2396] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260806 20:36:07.554467 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 148065us
I20260806 20:36:07.554001 17580 consensus_queue.cc:260] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.535561 17645 leader_election.cc:288] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.588801 17258 raft_consensus.cc:2781] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:36:07.589174 17155 raft_consensus.cc:3032] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:07.589634 17258 raft_consensus.cc:686] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Becoming Leader. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:07.556232 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354ed675014a3192fc7fdc9937479c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:07.519727 15178 raft_consensus.cc:2454] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:07.557070 17727 leader_election.cc:288] T a187485d6b5e4d9a9a9feafd48522726 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:07.530225 14470 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 10 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 9. Current term is 10. Ops: [9.5-9.5]
W20260806 20:36:07.601688 17761 log.cc:924] Time spent T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260806 20:36:07.602097 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:07.602921 15178 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:07.558451 15183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56934 (request call id 10) took 1351 ms. Trace:
I20260806 20:36:07.603416 15183 rpcz_store.cc:276] 0806 20:36:06.205097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.540859 (+1335762us) service_pool.cc:227] Handling call
0806 20:36:07.556391 (+ 15532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.603852 17580 raft_consensus.cc:3037] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:07.605072 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:07.605837 15183 raft_consensus.cc:3037] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:07.559335 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1817) took 2383 ms. Trace:
I20260806 20:36:07.608678 17577 leader_election.cc:288] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.608758 14547 rpcz_store.cc:276] 0806 20:36:05.175564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.325502 (+2149938us) service_pool.cc:227] Handling call
0806 20:36:07.396216 (+ 70714us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:36:07.440160 (+ 43944us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.559087 (+118927us) spinlock_profiling.cc:243] Waited 68.6 ms on lock 0x7b54000a8f50. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.559239 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":157869312}
I20260806 20:36:07.610129 14596 leader_election.cc:302] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:07.610786 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01e9209cd924024ae8e7e630b5accd3" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:07.612805 17457 leader_election.cc:288] T 500956bd1a3846a3a7b16c51ddecbc44 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.613385 14470 pending_rounds.cc:76] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Aborting all ops after (but not including) 3
I20260806 20:36:07.613739 14470 pending_rounds.cc:97] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:36:07.614003 14470 raft_consensus.cc:2866] T 08d2a67d79c54f588be9e64dce5b2202 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:36:07.614130 14441 leader_election.cc:302] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:07.613838 17613 raft_consensus.cc:504] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
W20260806 20:36:07.560596 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 206743us
I20260806 20:36:07.616050 15150 leader_election.cc:302] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.617065 17645 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:07.559376 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:07.571300 14473 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 8. Current term is 9. Ops: []
I20260806 20:36:07.620891 17683 raft_consensus.cc:2781] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:07.621178 14473 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 8. Current term is 9. Ops: []
I20260806 20:36:07.621538 17683 raft_consensus.cc:686] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 8 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:07.622882 14473 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P 92828dd6dbe74f29a04d4e702b502621 [term 5 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:07.624842 17258 consensus_queue.cc:237] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.571421 14442 leader_election.cc:302] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:07.626964 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5277702c39f453cb1063cf506183484" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:07.627270 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 77593us
W20260806 20:36:07.572050 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 171793us
I20260806 20:36:07.572571 17531 leader_election.cc:288] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:07.573776 15186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 675) took 1366 ms. Trace:
I20260806 20:36:07.630307 15186 rpcz_store.cc:276] 0806 20:36:06.206754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.317669 (+110915us) service_pool.cc:227] Handling call
0806 20:36:06.378523 (+ 60854us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:07.024361 (+645838us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.573218 (+548857us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b5400640d54. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:07.573568 (+ 350us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1412281216}
I20260806 20:36:07.574110 14595 leader_election.cc:302] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:07.575031 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.575256 17646 raft_consensus.cc:3032] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:36:07.576104 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1827) took 2259 ms. Trace:
I20260806 20:36:07.616145 17613 leader_election.cc:288] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.577986 17658 raft_consensus.cc:504] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.579210 17690 raft_consensus.cc:482] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:07.579756 17111 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:07.580686 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:07.581952 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:07.584216 17554 consensus_queue.cc:1046] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:36:07.585312 17663 consensus_queue.cc:237] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.567507 15180 rpcz_store.cc:276] 0806 20:36:06.423643 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.434288 (+1010645us) service_pool.cc:227] Handling call
0806 20:36:07.436022 (+ 1734us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.448860 (+ 12838us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.457497 (+ 8637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.655617 16844 raft_consensus.cc:3032] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:07.656002 16844 raft_consensus.cc:729] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 148bf2a11dba423ab364371f0458e871, State: Running, Role: LEADER
I20260806 20:36:07.656100 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:07.589668 17155 raft_consensus.cc:729] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:07.590595 14554 raft_consensus.cc:3037] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Advancing to term 9
W20260806 20:36:07.556761 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 96252us
I20260806 20:36:07.589658 14548 raft_consensus.cc:2454] T 568833ac14f74d16a95723bbfa721a96 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:36:07.611744 14547 raft_consensus.cc:3037] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:07.577076 14546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1826) took 2398 ms. Trace:
I20260806 20:36:07.631238 14471 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:36:07.636224 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1807) took 5369 ms. Trace:
I20260806 20:36:07.632752 17646 raft_consensus.cc:729] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:07.637404 14624 raft_consensus.cc:2454] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 7.
I20260806 20:36:07.640581 14555 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:36:07.640666 17352 raft_consensus.cc:2781] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 20:36:07.660058 14546 rpcz_store.cc:276] 0806 20:36:05.202761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.523907 (+2321146us) service_pool.cc:227] Handling call
0806 20:36:07.556238 (+ 32331us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.576676 (+ 20438us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:36:07.576706 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:07.576772 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:07.576804 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.576866 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.576986 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19837,"raft.queue_time_us":20118,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84}
I20260806 20:36:07.664882 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:07.666177 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:07.666474 14632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:07.666958 15181 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:07.661178 14551 rpcz_store.cc:276] 0806 20:36:02.266612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.393622 (+4127010us) service_pool.cc:227] Handling call
0806 20:36:06.419788 (+ 26166us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.400332 (+980544us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.628496 (+228164us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b54002d9bd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:07.636141 (+ 7645us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2245204864}
I20260806 20:36:07.668241 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:07.668944 14551 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:07.669693 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa7fe42a5543c3970808c8a8e8e45c" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.669333 17155 consensus_queue.cc:260] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.670373 14546 raft_consensus.cc:2454] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:07.670641 17155 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:07.671466 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:07.647562 17457 consensus_queue.cc:1046] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:36:07.648622 17332 raft_consensus.cc:2726] T 58bf5720d4f5403b877975863b1cd46c P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:36:07.648905 17658 leader_election.cc:288] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 election: Requested vote from peers df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.657688 15187 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:07.657153 17661 raft_consensus.cc:2726] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:07.663748 14628 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
W20260806 20:36:07.665078 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 55374.2us
I20260806 20:36:07.685339 14548 raft_consensus.cc:1226] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:07.672216 14546 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.658349 14633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae2b574a134bb781ba6c1a424f84a4" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:07.679543 15183 raft_consensus.cc:2454] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:07.681126 14474 raft_consensus.cc:2454] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:07.631460 14553 rpcz_store.cc:276] 0806 20:36:05.317151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.501417 (+2184266us) service_pool.cc:227] Handling call
0806 20:36:07.502009 (+ 592us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.502485 (+ 476us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:07.503033 (+ 548us) log.cc:842] Serialized 14 byte log entry
0806 20:36:07.548844 (+ 45811us) log.cc:842] Serialized 14 byte log entry
0806 20:36:07.575807 (+ 26963us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:07.575859 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:07.575897 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.575956 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.576052 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18527,"prepare.queue_time_us":438,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":77,"raft.queue_time_us":27464,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"thread_start_us":44727,"threads_started":1,"wal-append.queue_time_us":101179}
I20260806 20:36:07.645617 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:07.691828 14627 raft_consensus.cc:1226] T 8986697653e048048cd32eb27e4ae3d4 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 20:36:07.695797 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1828) took 2344 ms. Trace:
I20260806 20:36:07.695457 17646 consensus_queue.cc:260] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.696179 14548 rpcz_store.cc:276] 0806 20:36:05.349078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.684330 (+2335252us) service_pool.cc:227] Handling call
0806 20:36:07.685173 (+ 843us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.693146 (+ 7973us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.693270 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.696856 14444 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.697306 17646 raft_consensus.cc:3037] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:07.697886 15186 raft_consensus.cc:3037] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.691617 14518 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.622845 17683 consensus_queue.cc:237] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.702003 17661 consensus_queue.cc:1046] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:36:07.703109 17332 consensus_queue.cc:1046] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:36:07.703595 17661 raft_consensus.cc:3032] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:07.704059 17661 raft_consensus.cc:729] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:07.704569 17332 raft_consensus.cc:3032] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:07.720631 17332 raft_consensus.cc:729] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:07.720882 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.721695 14550 raft_consensus.cc:2454] T 07ca0af1acba4a7ea01bf15f5f887457 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:07.706177 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:07.721189 17661 consensus_queue.cc:260] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.722615 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.657217 16844 consensus_queue.cc:260] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.723363 14550 raft_consensus.cc:2454] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:07.723682 16844 raft_consensus.cc:3037] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:07.725070 14479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b00d6f3484ddbac8e7f8c2bdebba8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.725657 14479 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:07.678778 17690 raft_consensus.cc:504] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.730057 17690 leader_election.cc:288] T d2ced25ce9f1484081a8671550489f9b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.709224 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.730906 14555 raft_consensus.cc:2379] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 7: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:36:07.731122 14549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1824) took 2553 ms (client timeout 1657 ms). Trace:
W20260806 20:36:07.731477 14549 rpcz_store.cc:269] 0806 20:36:05.177993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:05.413849 (+235856us) service_pool.cc:227] Handling call
0806 20:36:07.730708 (+2316859us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540018ddd4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.730888 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2991380352}
I20260806 20:36:07.732106 14555 raft_consensus.cc:1226] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:07.732786 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1832) took 2205 ms. Trace:
I20260806 20:36:07.733043 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815d5557ccf4e569039960d0a8f24ba" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.733985 17661 raft_consensus.cc:3037] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:07.709052 17718 raft_consensus.cc:504] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.711120 17727 consensus_queue.cc:1046] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:36:07.703289 14518 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters:
I20260806 20:36:07.713116 14596 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:07.714910 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 67740.3us
I20260806 20:36:07.715446 17728 raft_consensus.cc:504] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.716277 17111 raft_consensus.cc:504] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:07.717340 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 143407us
I20260806 20:36:07.717991 17684 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:07.753978 14625 raft_consensus.cc:1226] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Rejecting Update request from peer 148bf2a11dba423ab364371f0458e871 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:07.722020 17332 consensus_queue.cc:260] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } attrs { replace: true } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:07.733083 14555 rpcz_store.cc:276] 0806 20:36:05.527377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.731532 (+2204155us) service_pool.cc:227] Handling call
0806 20:36:07.731892 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.732546 (+ 654us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.732716 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.724402 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.737279 17727 raft_consensus.cc:3032] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:36:07.750583 17718 leader_election.cc:288] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:07.753203 17728 leader_election.cc:288] T 87adab531acb4032af0c9a18dd9893e0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:07.719767 14477 raft_consensus.cc:2454] T 1e40012e131a4bfe9d61a3653f3298c6 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 6.
I20260806 20:36:07.738480 14521 leader_election.cc:302] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:07.690337 17352 raft_consensus.cc:686] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [term 10 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:07.759428 14549 raft_consensus.cc:1226] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:07.761368 15179 raft_consensus.cc:1226] T a53ea52d2ecf44bd87a1c23ea5f5f8df P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:07.764384 14595 leader_election.cc:302] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:07.755343 15186 raft_consensus.cc:2454] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:07.820015 15150 leader_election.cc:302] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:07.821278 14596 leader_election.cc:302] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:07.758443 14551 raft_consensus.cc:2454] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 9.
I20260806 20:36:07.828286 14478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1717) took 1064 ms. Trace:
I20260806 20:36:07.768860 14550 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 9.
I20260806 20:36:07.856528 14478 rpcz_store.cc:276] 0806 20:36:06.764042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.432114 (+668072us) service_pool.cc:227] Handling call
0806 20:36:07.828119 (+396005us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b54003f57d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.828231 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3613952}
I20260806 20:36:07.767271 14554 raft_consensus.cc:2454] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 9.
I20260806 20:36:07.835570 14469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54502 (request call id 1802) took 1357 ms. Trace:
I20260806 20:36:07.858469 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54498 (request call id 1743) took 1180 ms. Trace:
W20260806 20:36:07.822999 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 142820us
I20260806 20:36:07.817296 14518 leader_election.cc:302] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.829006 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.862429 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1722) took 2039 ms. Trace:
I20260806 20:36:07.862789 14549 rpcz_store.cc:276] 0806 20:36:05.823277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.732316 (+1909039us) service_pool.cc:227] Handling call
0806 20:36:07.754741 (+ 22425us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.855470 (+100729us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.862370 (+ 6900us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:07.816836 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 51196.7us
I20260806 20:36:07.859046 14554 rpcz_store.cc:276] 0806 20:36:06.678357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.731374 (+ 53017us) service_pool.cc:227] Handling call
0806 20:36:07.555987 (+824613us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b5c000f618c. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b6e3a361 00007fb9b6fd41f9 00007fb9b8434ac6 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:07.858398 (+302411us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181257088}
I20260806 20:36:06.792798 13282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:36:07.866250 14548 raft_consensus.cc:1226] T fe354ed675014a3192fc7fdc9937479c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 8. Current term is 9. Ops: []
I20260806 20:36:07.865878 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.814702 14553 raft_consensus.cc:1226] T 05f645749c8846cf8c2f4dd196506e4d P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:07.867667 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1830) took 2491 ms. Trace:
I20260806 20:36:07.867875 14553 rpcz_store.cc:276] 0806 20:36:05.376310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.693436 (+2317126us) service_pool.cc:227] Handling call
0806 20:36:07.814378 (+120942us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.867461 (+ 53083us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.867627 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.868732 14596 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:07.827591 14631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1663) took 1190 ms. Trace:
I20260806 20:36:07.858304 14469 rpcz_store.cc:276] 0806 20:36:06.478248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.478466 (+ 218us) service_pool.cc:227] Handling call
0806 20:36:06.478926 (+ 460us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.307042 (+828116us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.835194 (+528152us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54002166d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:07.835474 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1899736320}
I20260806 20:36:07.873967 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:07.763864 17111 leader_election.cc:288] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:07.878132 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f41753baced49a1997940dfc28b2e00" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:07.879235 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 188827us
I20260806 20:36:07.870285 14631 rpcz_store.cc:276] 0806 20:36:06.637374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.327369 (+689995us) service_pool.cc:227] Handling call
0806 20:36:07.328329 (+ 960us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.466691 (+138362us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:36:07.559659 (+ 92968us) log.cc:842] Serialized 14 byte log entry
0806 20:36:07.634497 (+ 74838us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:07.634563 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:07.644152 (+ 9589us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.827376 (+183224us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540009f954. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:07.827429 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:07.827538 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":192751,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":42116,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":14794,"spinlock_wait_cycles":300750720,"thread_start_us":124717,"threads_started":2,"wal-append.queue_time_us":139792}
I20260806 20:36:07.820567 17602 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:36:07.881857 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 181532us
I20260806 20:36:07.765758 17727 raft_consensus.cc:729] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:07.881752 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:07.885617 14551 raft_consensus.cc:2454] T eb32d3aa416443529139823cf8e2d00b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 6.
I20260806 20:36:07.874163 17529 raft_consensus.cc:2781] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:07.822777 17688 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:36:07.818507 17457 consensus_queue.cc:1225] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer 148bf2a11dba423ab364371f0458e871 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260806 20:36:07.897749 17684 raft_consensus.cc:2781] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:07.898504 17722 raft_consensus.cc:2726] T 3694b5a404a94fbea69b831722e2d3e4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:07.899991 17486 raft_consensus.cc:2726] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:07.902989 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4248ec88bd4b44e990938a74fe3c4fdc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.910908 14444 leader_election.cc:302] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621; no voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.911659 14547 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:07.913241 17685 raft_consensus.cc:2726] T db55591983ce42868e62df33ee1b56be P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:07.891925 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.914789 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:07.920228 14469 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 20:36:07.918011 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:07.920770 17688 raft_consensus.cc:504] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.916268 17638 raft_consensus.cc:2726] T 90e4ac7b6cd245aead7963fa7cbc5d2e P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:36:07.922860 17688 leader_election.cc:288] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:07.924921 14441 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:07.925355 17685 consensus_queue.cc:1046] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:36:07.926019 17459 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:36:07.926589 15181 raft_consensus.cc:2454] T dc270596f078400e9e1b181a9706b516 P 148bf2a11dba423ab364371f0458e871 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 9.
I20260806 20:36:07.926613 17459 raft_consensus.cc:686] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:07.932021 14441 leader_election.cc:302] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:07.924439 15187 raft_consensus.cc:2454] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 7.
I20260806 20:36:07.926807 17690 raft_consensus.cc:3032] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:36:07.879117 17376 raft_consensus.cc:2781] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:07.952518 14475 raft_consensus.cc:2454] T 4248ec88bd4b44e990938a74fe3c4fdc P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148bf2a11dba423ab364371f0458e871 in term 7.
I20260806 20:36:07.955299 14441 leader_election.cc:302] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:07.884838 17602 raft_consensus.cc:482] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:07.957158 14479 raft_consensus.cc:2454] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:07.954710 14442 leader_election.cc:302] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:07.973093 14549 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:07.973869 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1838) took 1543 ms. Trace:
I20260806 20:36:07.974260 14549 rpcz_store.cc:276] 0806 20:36:06.430171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.890874 (+1460703us) service_pool.cc:227] Handling call
0806 20:36:07.972837 (+ 81963us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.973590 (+ 753us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.973800 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.975267 14478 raft_consensus.cc:2379] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 7: Already voted for candidate 92828dd6dbe74f29a04d4e702b502621 in this term.
I20260806 20:36:07.976902 17684 raft_consensus.cc:686] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:07.978196 17684 consensus_queue.cc:237] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.952131 17690 raft_consensus.cc:729] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:07.982105 17690 consensus_queue.cc:260] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:07.983803 17690 raft_consensus.cc:3037] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.989389 14553 raft_consensus.cc:1226] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 20:36:07.990195 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1726) took 2165 ms. Trace:
I20260806 20:36:07.990592 14553 rpcz_store.cc:276] 0806 20:36:05.824553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.880071 (+2055518us) service_pool.cc:227] Handling call
0806 20:36:07.989042 (+108971us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:07.989913 (+ 871us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.990139 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:08.037456 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 82263.6us
I20260806 20:36:08.044382 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1730) took 1543 ms. Trace:
I20260806 20:36:07.894568 17529 raft_consensus.cc:482] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:07.928210 17737 raft_consensus.cc:2741] T d01e9209cd924024ae8e7e630b5accd3 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260806 20:36:07.958945 14554 raft_consensus.cc:3037] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:07.962170 14479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54500 (request call id 1725) took 1194 ms. Trace:
I20260806 20:36:08.054224 14479 rpcz_store.cc:276] 0806 20:36:06.767968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.335221 (+567253us) service_pool.cc:227] Handling call
0806 20:36:07.724905 (+389684us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5800003398. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b040ff11 00007fb9b0593851 00007fb9b0593702 00007fb9b9483b09 00007fb9b83bf730 00007fb9b848579b 00007fb9b8485821 00007fb9b84351b0 00007fb9b8434a94 00007fb9b1b1cc04 00007fb9b1b1cb92
0806 20:36:07.961992 (+237087us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28084608}
W20260806 20:36:07.958302 14521 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 71365.4us
I20260806 20:36:08.044909 14551 rpcz_store.cc:276] 0806 20:36:06.500985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.915277 (+1414292us) service_pool.cc:227] Handling call
0806 20:36:07.915614 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.915991 (+ 377us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:36:07.918588 (+ 2597us) log.cc:842] Serialized 14 byte log entry
0806 20:36:08.043998 (+125410us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:08.044062 (+ 64us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:08.044121 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.044188 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.044322 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":154,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":86388,"threads_started":1}
W20260806 20:36:08.050941 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 170576us
W20260806 20:36:08.051988 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 71342.5us
I20260806 20:36:07.958979 14546 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
W20260806 20:36:08.062670 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 245544us
W20260806 20:36:08.063611 15150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 243699us
I20260806 20:36:07.954841 17376 raft_consensus.cc:482] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
W20260806 20:36:08.071223 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 307008us
W20260806 20:36:08.070616 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 187804us
I20260806 20:36:08.070580 14549 raft_consensus.cc:1226] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:08.074532 14549 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1733) took 1572 ms. Trace:
I20260806 20:36:08.074882 14549 rpcz_store.cc:276] 0806 20:36:06.501630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.045948 (+1544318us) service_pool.cc:227] Handling call
0806 20:36:08.070332 (+ 24384us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.073805 (+ 3473us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.074476 (+ 671us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:07.957515 17685 raft_consensus.cc:2781] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:08.086395 14442 leader_election.cc:302] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:08.084888 14521 leader_election.cc:302] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
I20260806 20:36:07.959695 17602 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:08.055771 17529 raft_consensus.cc:3037] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:08.060598 15182 raft_consensus.cc:1226] T 7f41753baced49a1997940dfc28b2e00 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 20:36:08.064900 17332 raft_consensus.cc:3037] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:08.096856 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:08.097293 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:08.098115 14518 leader_election.cc:302] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:08.071823 17376 raft_consensus.cc:3037] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:08.081784 17457 raft_consensus.cc:2781] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:07.955412 17258 consensus_queue.cc:1046] T 87adab531acb4032af0c9a18dd9893e0 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.133s
I20260806 20:36:08.100701 15151 leader_election.cc:302] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:07.928006 17459 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.044700 17155 consensus_queue.cc:1046] T cf24df4ca64846378f216b30e63e27f0 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 20:36:08.109936 14554 raft_consensus.cc:2454] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 8.
I20260806 20:36:08.110904 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:08.116591 14442 leader_election.cc:302] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters: df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:08.116894 17679 consensus_queue.cc:1046] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260806 20:36:08.117969 17679 raft_consensus.cc:3032] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:08.118094 14470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1714) took 1355 ms. Trace:
I20260806 20:36:08.118371 17679 raft_consensus.cc:729] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:08.122458 14554 raft_consensus.cc:2379] T 8844659c5dd24572bab900a68fc14df6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148bf2a11dba423ab364371f0458e871 in current term 8: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
I20260806 20:36:08.123863 17680 raft_consensus.cc:2781] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:36:08.146571 17542 raft_consensus.cc:2781] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader election won for term 9
W20260806 20:36:08.147254 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 52516.5us
I20260806 20:36:08.118584 14470 rpcz_store.cc:276] 0806 20:36:06.763137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.612592 (+849455us) service_pool.cc:227] Handling call
0806 20:36:07.613008 (+ 416us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:36:07.650921 (+ 37913us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:07.650951 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:36:07.658749 (+ 7798us) log.cc:842] Serialized 52 byte log entry
0806 20:36:07.872565 (+213816us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:36:07.872594 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:07.872640 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:36:08.117836 (+245196us) raft_consensus.cc:1671] finished
0806 20:36:08.117856 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.118013 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15677,"raft.queue_time_us":28570,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"thread_start_us":178106,"threads_started":1,"wal-append.queue_time_us":276419}
I20260806 20:36:08.112838 17486 raft_consensus.cc:2781] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:08.108465 17457 raft_consensus.cc:482] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
W20260806 20:36:08.164336 15151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 64685.3us
W20260806 20:36:08.121881 15178 consensus_meta.cc:287] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871: Time spent flushing consensus metadata: real 0.518s user 0.010s sys 0.000s
I20260806 20:36:08.168255 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:08.174290 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 53822us
I20260806 20:36:08.069790 17352 consensus_queue.cc:237] T 6c8aa43086db48d2821816b1223056dd P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.179611 15151 leader_election.cc:302] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:08.181425 17393 consensus_queue.cc:1046] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260806 20:36:08.119437 17679 consensus_queue.cc:260] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.185611 17679 raft_consensus.cc:3037] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:08.177299 17457 raft_consensus.cc:3037] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:08.106287 17688 consensus_queue.cc:1046] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260806 20:36:08.191954 17505 consensus_queue.cc:1046] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 20:36:08.207290 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 62135.9us
I20260806 20:36:08.147292 17703 raft_consensus.cc:2726] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:08.213228 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53ea52d2ecf44bd87a1c23ea5f5f8df" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:08.215929 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:08.164216 17680 raft_consensus.cc:686] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:08.216670 15186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986697653e048048cd32eb27e4ae3d4" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:08.218325 15187 raft_consensus.cc:2396] T 8986697653e048048cd32eb27e4ae3d4 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 20:36:08.222232 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8844659c5dd24572bab900a68fc14df6" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
W20260806 20:36:08.222446 17775 log.cc:924] Time spent T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 20:36:08.222949 15183 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:08.223508 17393 raft_consensus.cc:3032] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 LEADER]: Stepping down as leader of term 9
I20260806 20:36:08.224049 17393 raft_consensus.cc:729] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:08.224673 17787 raft_consensus.cc:2781] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:36:08.225234 17787 raft_consensus.cc:482] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:36:08.176095 17542 raft_consensus.cc:686] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [term 9 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:08.211673 17486 raft_consensus.cc:686] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 LEADER]: Becoming Leader. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:08.233372 17542 consensus_queue.cc:237] T dc270596f078400e9e1b181a9706b516 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.233853 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:08.234366 17486 consensus_queue.cc:237] T 8986697653e048048cd32eb27e4ae3d4 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.225764 17555 raft_consensus.cc:3032] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:08.237001 17555 raft_consensus.cc:729] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:08.237154 14595 leader_election.cc:302] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:36:08.217880 17255 raft_consensus.cc:482] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:08.239070 17750 consensus_queue.cc:1046] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:36:08.238000 17555 consensus_queue.cc:260] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.239755 17255 raft_consensus.cc:504] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.225272 17393 consensus_queue.cc:260] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.241472 17393 raft_consensus.cc:3037] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:36:08.241990 17255 leader_election.cc:288] T 4248ec88bd4b44e990938a74fe3c4fdc P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:08.242651 17255 raft_consensus.cc:482] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:08.243141 17255 raft_consensus.cc:504] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.228339 17530 raft_consensus.cc:3032] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:36:08.244803 17530 raft_consensus.cc:729] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:08.245702 17530 consensus_queue.cc:260] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.240355 17555 raft_consensus.cc:3037] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:08.251463 15183 raft_consensus.cc:2454] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in term 8.
W20260806 20:36:08.217502 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 117735us
I20260806 20:36:08.278556 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d79d20363456d80521feddfec49bc" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:08.179608 15178 raft_consensus.cc:2454] T d01e9209cd924024ae8e7e630b5accd3 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 5.
I20260806 20:36:08.285490 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1735) took 1783 ms. Trace:
I20260806 20:36:08.288928 14551 rpcz_store.cc:276] 0806 20:36:06.502067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.074143 (+1572076us) service_pool.cc:227] Handling call
0806 20:36:08.084899 (+ 10756us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.085217 (+ 318us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:36:08.085548 (+ 331us) log.cc:842] Serialized 14 byte log entry
0806 20:36:08.284998 (+199450us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:08.285078 (+ 80us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:08.285152 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.285246 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.285417 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15215,"prepare.queue_time_us":658,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":4412,"thread_start_us":166817,"threads_started":1}
I20260806 20:36:08.321167 15151 leader_election.cc:302] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:08.339066 14627 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:36:08.252837 17787 raft_consensus.cc:3037] T 568833ac14f74d16a95723bbfa721a96 P 148bf2a11dba423ab364371f0458e871 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:08.361009 14550 raft_consensus.cc:1226] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:08.342234 17530 raft_consensus.cc:3037] T 05f645749c8846cf8c2f4dd196506e4d P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:36:08.363536 14595 leader_election.cc:302] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 92828dd6dbe74f29a04d4e702b502621
W20260806 20:36:08.371145 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 268836us
I20260806 20:36:08.378050 14554 raft_consensus.cc:1226] T 56e9ab616aca4e6bb97d457694148205 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260806 20:36:08.378974 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1741) took 1763 ms. Trace:
I20260806 20:36:08.379379 14554 rpcz_store.cc:276] 0806 20:36:06.615226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.348515 (+1733289us) service_pool.cc:227] Handling call
0806 20:36:08.377730 (+ 29215us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:36:08.378624 (+ 894us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.378907 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.288051 17727 consensus_queue.cc:260] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.390563 14551 raft_consensus.cc:1226] T 1e40012e131a4bfe9d61a3653f3298c6 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 5. Current term is 6. Ops: []
I20260806 20:36:08.391492 14551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1740) took 1842 ms. Trace:
I20260806 20:36:08.391939 14551 rpcz_store.cc:276] 0806 20:36:06.549361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.341706 (+1792345us) service_pool.cc:227] Handling call
0806 20:36:08.381136 (+ 39430us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.391145 (+ 10009us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.391411 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.346920 17688 consensus_queue.cc:1035] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260806 20:36:08.370735 17661 raft_consensus.cc:3032] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:36:08.393863 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8aa43086db48d2821816b1223056dd" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:08.404908 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1660) took 1768 ms. Trace:
I20260806 20:36:08.405452 14629 rpcz_store.cc:276] 0806 20:36:06.636558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.191030 (+554472us) service_pool.cc:227] Handling call
0806 20:36:07.474990 (+283960us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.541077 (+ 66087us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.403587 (+862510us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400313150. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:08.404844 (+ 1257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":647655168}
I20260806 20:36:08.402563 17754 raft_consensus.cc:2781] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:08.398142 17600 raft_consensus.cc:2781] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:36:08.415452 14551 raft_consensus.cc:3032] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:08.431111 14551 raft_consensus.cc:729] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8e0cfd35f2994f91aaae46c25c6d6bff, State: Running, Role: LEADER
I20260806 20:36:08.399627 17661 raft_consensus.cc:729] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:08.445853 17661 consensus_queue.cc:260] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
W20260806 20:36:08.446569 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 126880us
I20260806 20:36:08.447237 17661 raft_consensus.cc:3037] T fcae2b574a134bb781ba6c1a424f84a4 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:08.447980 14595 leader_election.cc:302] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:36:08.448796 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e455c79bfc43e9ac867cf6719e289c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:08.449374 17688 raft_consensus.cc:2726] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 20:36:08.449596 14595 leader_election.cc:398] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8.
I20260806 20:36:08.449970 14595 leader_election.cc:401] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 20:36:08.450009 17505 raft_consensus.cc:2781] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:36:08.450685 17505 raft_consensus.cc:686] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [term 5 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:08.421172 17258 raft_consensus.cc:2726] T 8844659c5dd24572bab900a68fc14df6 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:08.452943 17728 raft_consensus.cc:2726] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8.
W20260806 20:36:08.454165 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 337753us
W20260806 20:36:08.409116 17580 consensus_meta.cc:287] T 370a97aea0534403aeb11fe85b5a1ca9 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.805s user 0.014s sys 0.005s
I20260806 20:36:08.453886 17258 raft_consensus.cc:2726] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:36:08.463084 17258 consensus_queue.cc:1035] T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:36:08.473500 15183 raft_consensus.cc:1226] T 3044f79ab3f74f788d8a263e141674a8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260806 20:36:08.411386 17255 leader_election.cc:288] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:08.484625 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf24df4ca64846378f216b30e63e27f0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:08.454200 17727 raft_consensus.cc:3037] T 8844659c5dd24572bab900a68fc14df6 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:08.497524 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:08.498061 14627 raft_consensus.cc:1226] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Rejecting Update request from peer 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:08.500808 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb469b8694714b21bdf2c86de7aa2dad" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.501593 14475 raft_consensus.cc:2454] T fb469b8694714b21bdf2c86de7aa2dad P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 7.
I20260806 20:36:08.474697 14551 consensus_queue.cc:260] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.458850 17600 raft_consensus.cc:686] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [term 9 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:08.451901 17505 consensus_queue.cc:237] T 8d5ff5043e9e4bdbb65a2354b475a2c1 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.500883 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.461874 17685 raft_consensus.cc:686] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [term 8 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:08.459339 17754 raft_consensus.cc:686] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:08.563092 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
W20260806 20:36:08.562621 15153 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 218939us
W20260806 20:36:08.572317 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 66008.5us
I20260806 20:36:08.511109 17688 raft_consensus.cc:482] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:08.444156 17599 consensus_queue.cc:1046] T fe354ed675014a3192fc7fdc9937479c P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.268s
I20260806 20:36:08.573709 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.576126 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef11d9fbd5a449aca4e91c72a6e73290" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:08.585949 14470 raft_consensus.cc:2396] T ef11d9fbd5a449aca4e91c72a6e73290 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 20:36:08.514006 17783 raft_consensus.cc:2781] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:36:08.595615 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.611667 15187 raft_consensus.cc:1261] T 6a9d79d20363456d80521feddfec49bc P 148bf2a11dba423ab364371f0458e871 [term 6 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:36:08.594797 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1731) took 2080 ms. Trace:
I20260806 20:36:08.573144 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:08.443064 17680 consensus_queue.cc:237] T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.628911 15185 raft_consensus.cc:1226] T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 5 LEADER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5. Ops: []
I20260806 20:36:08.618700 14473 raft_consensus.cc:1226] T cfcc7f43dcc54dc48f61c69d3d4f99a3 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:08.613920 14596 leader_election.cc:302] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, df1245ecd2034e11ae6cf688c1c10a4d; no voters:
W20260806 20:36:08.620910 15184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59384 (request call id 671) took 2415 ms (client timeout 3218 ms). Trace:
I20260806 20:36:08.625000 17600 consensus_queue.cc:237] T fcae2b574a134bb781ba6c1a424f84a4 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.518430 17529 raft_consensus.cc:504] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.643332 14627 raft_consensus.cc:1261] T ef11d9fbd5a449aca4e91c72a6e73290 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:36:08.663061 15178 raft_consensus.cc:1261] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 20:36:08.644793 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bd3d0855da421aac4540ae1bfbb8f3" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.639384 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.632910 14441 leader_election.cc:302] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:08.638176 15187 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 20:36:08.615195 14547 rpcz_store.cc:276] 0806 20:36:06.501213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.959339 (+1458126us) service_pool.cc:227] Handling call
0806 20:36:07.996997 (+ 37658us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.997338 (+ 341us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:36:07.998247 (+ 909us) log.cc:842] Serialized 14 byte log entry
0806 20:36:08.167299 (+169052us) log.cc:842] Serialized 14 byte log entry
0806 20:36:08.581396 (+414097us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:08.581466 (+ 70us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:08.581530 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.581616 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.581763 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26987,"prepare.queue_time_us":26113,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":82,"raft.queue_time_us":84373,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"spinlock_wait_cycles":23723520,"thread_start_us":510388,"threads_started":2,"wal-append.queue_time_us":193211}
I20260806 20:36:08.638564 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e391f934d63c4ad788870d7abc39a79c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.587215 17649 raft_consensus.cc:3032] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:36:08.641824 17376 consensus_meta.cc:287] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.521s user 0.011s sys 0.000s
W20260806 20:36:08.669039 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 182379us
I20260806 20:36:08.771857 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:08.764492 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1841) took 1981 ms. Trace:
I20260806 20:36:08.775291 14595 leader_election.cc:302] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:36:08.808353 14475 raft_consensus.cc:2354] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 7. Current term is 8.
I20260806 20:36:08.810513 14442 leader_election.cc:302] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:08.812202 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1734) took 2310 ms. Trace:
I20260806 20:36:08.813131 14553 rpcz_store.cc:276] 0806 20:36:06.501865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.046316 (+1544451us) service_pool.cc:227] Handling call
0806 20:36:08.089572 (+ 43256us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.095596 (+ 6024us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:36:08.317907 (+222311us) log.cc:842] Serialized 14 byte log entry
0806 20:36:08.811709 (+493802us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:08.811785 (+ 76us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:36:08.811851 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.811960 (+ 109us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.812128 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":30188,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":5578,"raft.queue_time_us":61110,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":48,"thread_start_us":654313,"threads_started":3,"wal-append.queue_time_us":412579}
W20260806 20:36:08.663021 15184 rpcz_store.cc:269] 0806 20:36:06.205413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.278261 (+ 72848us) service_pool.cc:227] Handling call
0806 20:36:08.568329 (+2290068us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b54006f04d4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:08.620651 (+ 52322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":790450944}
I20260806 20:36:08.771723 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9ab616aca4e6bb97d457694148205" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:08.818424 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 185643us
W20260806 20:36:08.754719 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 140918us
I20260806 20:36:08.820164 14441 leader_election.cc:302] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621; no voters:
W20260806 20:36:08.773419 17393 consensus_meta.cc:287] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.532s user 0.011s sys 0.001s
I20260806 20:36:08.808192 14554 rpcz_store.cc:276] 0806 20:36:06.751077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.404285 (+1653208us) service_pool.cc:227] Handling call
0806 20:36:08.443295 (+ 39010us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.695226 (+251931us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:08.695278 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:08.732436 (+ 37158us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:08.732516 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.732597 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.732771 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":181044,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":246203,"threads_started":1}
I20260806 20:36:08.857728 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87adab531acb4032af0c9a18dd9893e0" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:08.797286 17155 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.012s sys 0.009s
I20260806 20:36:08.797428 17783 raft_consensus.cc:686] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [term 9 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:08.870882 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef4a42f075d4ee1ac14ba3159109a18" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.773669 17491 raft_consensus.cc:482] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:08.870288 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38960b1dd9245a8ba85fea601b09e94" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:08.879190 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bf5720d4f5403b877975863b1cd46c" candidate_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:08.881758 14470 raft_consensus.cc:2454] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 6.
I20260806 20:36:08.879508 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5ff5043e9e4bdbb65a2354b475a2c1" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:08.806924 17649 raft_consensus.cc:729] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:08.734575 17255 raft_consensus.cc:482] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:08.888546 14478 raft_consensus.cc:1261] T f5277702c39f453cb1063cf506183484 P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260806 20:36:08.885584 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 75232.5us
I20260806 20:36:08.891681 14630 raft_consensus.cc:2396] T 6c8aa43086db48d2821816b1223056dd P df1245ecd2034e11ae6cf688c1c10a4d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260806 20:36:08.636575 17646 consensus_queue.cc:1046] T 3044f79ab3f74f788d8a263e141674a8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.240s
W20260806 20:36:08.894670 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 119485us
I20260806 20:36:08.895706 14595 leader_election.cc:302] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:08.630086 17684 consensus_queue.cc:1046] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.250s
W20260806 20:36:08.871389 17155 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.015s sys 0.015s
I20260806 20:36:08.752362 15181 raft_consensus.cc:2454] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1245ecd2034e11ae6cf688c1c10a4d in term 5.
I20260806 20:36:08.628391 17457 raft_consensus.cc:504] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.871484 17683 raft_consensus.cc:2781] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:36:08.753261 17679 consensus_queue.cc:1035] T ef11d9fbd5a449aca4e91c72a6e73290 P 8e0cfd35f2994f91aaae46c25c6d6bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.135s
I20260806 20:36:08.902177 14631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a9c839b021436dac626c7630f18f8e" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:08.911330 14470 raft_consensus.cc:1261] T bef4a42f075d4ee1ac14ba3159109a18 P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Refusing update from remote peer 8e0cfd35f2994f91aaae46c25c6d6bff: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 20:36:08.914860 14554 raft_consensus.cc:1261] T fefa7fe42a5543c3970808c8a8e8e45c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 20:36:08.915485 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1843) took 1859 ms. Trace:
I20260806 20:36:08.915800 14554 rpcz_store.cc:276] 0806 20:36:07.056018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.876758 (+1820740us) service_pool.cc:227] Handling call
0806 20:36:08.909774 (+ 33016us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.915254 (+ 5480us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.915435 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.637079 17685 consensus_queue.cc:237] T 3044f79ab3f74f788d8a263e141674a8 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.632665 17688 raft_consensus.cc:504] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } attrs { replace: true } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.874372 17505 raft_consensus.cc:2781] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:08.797705 17663 consensus_queue.cc:1046] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.098s
I20260806 20:36:08.922732 17783 consensus_queue.cc:237] T fe354ed675014a3192fc7fdc9937479c P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.923530 14631 raft_consensus.cc:3037] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:36:08.912158 14475 raft_consensus.cc:1226] T 457a5ba876d14681a3fa2338fd9f6c4e P 92828dd6dbe74f29a04d4e702b502621 [term 8 LEADER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: []
I20260806 20:36:08.924625 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1746) took 2139 ms. Trace:
I20260806 20:36:08.925206 14627 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:36:08.818480 17376 raft_consensus.cc:504] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.934667 14553 raft_consensus.cc:1226] T 6c8aa43086db48d2821816b1223056dd P 8e0cfd35f2994f91aaae46c25c6d6bff [term 10 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 9. Current term is 10. Ops: [9.5-9.5]
W20260806 20:36:08.918265 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 203435us
I20260806 20:36:08.920287 17393 raft_consensus.cc:2781] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:08.941280 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500956bd1a3846a3a7b16c51ddecbc44" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:08.925028 14547 rpcz_store.cc:276] 0806 20:36:06.785423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.807347 (+2021924us) service_pool.cc:227] Handling call
0806 20:36:08.899761 (+ 92414us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.924255 (+ 24494us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:08.924292 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:08.924334 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:08.924375 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.924439 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.924568 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.935482 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1842) took 1905 ms. Trace:
I20260806 20:36:08.936769 14479 raft_consensus.cc:1226] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260806 20:36:08.921484 17661 raft_consensus.cc:2726] T cf24df4ca64846378f216b30e63e27f0 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:36:08.942373 14553 rpcz_store.cc:276] 0806 20:36:07.029786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.825647 (+1795861us) service_pool.cc:227] Handling call
0806 20:36:08.918963 (+ 93316us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:36:08.935159 (+ 16196us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.935414 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.964993 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187485d6b5e4d9a9a9feafd48522726" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
W20260806 20:36:08.965583 17555 consensus_meta.cc:287] T 67e455c79bfc43e9ac867cf6719e289c P 92828dd6dbe74f29a04d4e702b502621: Time spent flushing consensus metadata: real 0.693s user 0.009s sys 0.001s
I20260806 20:36:08.965821 14555 raft_consensus.cc:2379] T a187485d6b5e4d9a9a9feafd48522726 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 8: Already voted for candidate 8e0cfd35f2994f91aaae46c25c6d6bff in this term.
W20260806 20:36:08.934886 17602 consensus_meta.cc:287] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 0.820s user 0.006s sys 0.005s
I20260806 20:36:08.966389 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54508 (request call id 1851) took 1209 ms. Trace:
I20260806 20:36:08.966765 14555 rpcz_store.cc:276] 0806 20:36:07.756632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.964697 (+1208065us) service_pool.cc:227] Handling call
0806 20:36:08.966333 (+ 1636us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.966550 17602 raft_consensus.cc:504] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.947799 17788 raft_consensus.cc:482] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:36:08.970934 17788 raft_consensus.cc:504] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.973551 17788 leader_election.cc:288] T d8a0e4079ebe47aaa3c275ed7f559ab8 P 148bf2a11dba423ab364371f0458e871 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:08.948725 17792 raft_consensus.cc:2781] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:08.951604 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3044f79ab3f74f788d8a263e141674a8" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
I20260806 20:36:08.974647 17792 raft_consensus.cc:482] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df1245ecd2034e11ae6cf688c1c10a4d)
I20260806 20:36:08.975154 17792 raft_consensus.cc:3037] T eb32d3aa416443529139823cf8e2d00b P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:08.976963 17505 raft_consensus.cc:686] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 LEADER]: Becoming Leader. State: Replica: df1245ecd2034e11ae6cf688c1c10a4d, State: Running, Role: LEADER
I20260806 20:36:08.977619 14551 raft_consensus.cc:3037] T 5b527f3bc7bb43f88f1b17c445c3c0d0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:08.978034 17505 consensus_queue.cc:237] T 56e9ab616aca4e6bb97d457694148205 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.955178 17529 leader_election.cc:288] T 07ca0af1acba4a7ea01bf15f5f887457 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:08.956139 14518 leader_election.cc:302] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 148bf2a11dba423ab364371f0458e871, 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:08.981032 14554 raft_consensus.cc:1226] T fcae2b574a134bb781ba6c1a424f84a4 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 9 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 8. Current term is 9. Ops: []
I20260806 20:36:08.981742 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1845) took 1648 ms. Trace:
I20260806 20:36:08.982417 14518 leader_election.cc:302] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:08.982643 14554 rpcz_store.cc:276] 0806 20:36:07.333160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.927027 (+1593867us) service_pool.cc:227] Handling call
0806 20:36:08.980849 (+ 53822us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.981487 (+ 638us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.981681 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.984301 17808 raft_consensus.cc:2726] T a94bd288a57145e8b254611b1dd82586 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:08.984987 17679 raft_consensus.cc:2726] T ac95ea9d55914ec6912deffb9a91d8f8 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:08.985193 14547 raft_consensus.cc:1261] T 289df1801e0045beb52c1aaaf43e73f0 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:36:08.985724 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 434) took 1556 ms. Trace:
I20260806 20:36:08.986022 14547 rpcz_store.cc:276] 0806 20:36:07.429425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.942226 (+1512801us) service_pool.cc:227] Handling call
0806 20:36:08.984810 (+ 42584us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.985516 (+ 706us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.985680 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.984351 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1753) took 1335 ms. Trace:
I20260806 20:36:08.986735 14554 rpcz_store.cc:276] 0806 20:36:07.648831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.983512 (+1334681us) service_pool.cc:227] Handling call
0806 20:36:08.983806 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.984045 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:08.984069 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:08.984120 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:36:08.984148 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.984196 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.984282 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.987252 14628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38500 (request call id 1656) took 2352 ms. Trace:
I20260806 20:36:08.987728 14628 rpcz_store.cc:276] 0806 20:36:06.635296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.054284 (+418988us) service_pool.cc:227] Handling call
0806 20:36:07.054744 (+ 460us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.703734 (+648990us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.986992 (+1283258us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54002c2254. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:08.987197 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1379876224}
I20260806 20:36:08.988372 14554 raft_consensus.cc:1226] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:08.989089 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1756) took 1279 ms. Trace:
I20260806 20:36:08.964257 14553 raft_consensus.cc:1261] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:36:08.990552 14553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1748) took 1487 ms. Trace:
I20260806 20:36:08.990849 14471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54500 (request call id 1723) took 2223 ms. Trace:
I20260806 20:36:08.989373 14554 rpcz_store.cc:276] 0806 20:36:07.709859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.987913 (+1278054us) service_pool.cc:227] Handling call
0806 20:36:08.988176 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.988828 (+ 652us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.989038 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:08.981683 14595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 83675.8us
I20260806 20:36:08.979552 14548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1831) took 3536 ms. Trace:
I20260806 20:36:08.992115 14518 leader_election.cc:302] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff; no voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:08.998566 14595 leader_election.cc:302] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1245ecd2034e11ae6cf688c1c10a4d; no voters: 148bf2a11dba423ab364371f0458e871, 92828dd6dbe74f29a04d4e702b502621
I20260806 20:36:08.998584 14548 rpcz_store.cc:276] 0806 20:36:05.462311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.697010 (+2234699us) service_pool.cc:227] Handling call
0806 20:36:07.858182 (+161172us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.870715 (+ 12533us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.979335 (+1108620us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400355a50. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:08.979486 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369585024}
I20260806 20:36:08.953738 14478 raft_consensus.cc:1226] T 119ce96f4f7f49cfa853221635714edc P 92828dd6dbe74f29a04d4e702b502621 [term 7 FOLLOWER]: Rejecting Update request from peer df1245ecd2034e11ae6cf688c1c10a4d for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260806 20:36:08.892839 17491 raft_consensus.cc:504] T 6b370f85a8b944198972759bc1eb8061 P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } } peers { permanent_uuid: "148bf2a11dba423ab364371f0458e871" member_type: VOTER last_known_addr { host: "127.12.248.132" port: 42399 } attrs { promote: false } }
I20260806 20:36:08.987286 14547 raft_consensus.cc:1226] T fb469b8694714b21bdf2c86de7aa2dad P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Rejecting Update request from peer 92828dd6dbe74f29a04d4e702b502621 for earlier term 6. Current term is 7. Ops: []
I20260806 20:36:08.954407 17457 leader_election.cc:288] T fefa7fe42a5543c3970808c8a8e8e45c P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), df1245ecd2034e11ae6cf688c1c10a4d (127.12.248.131:45209)
I20260806 20:36:08.955176 14629 raft_consensus.cc:2454] T 500956bd1a3846a3a7b16c51ddecbc44 P df1245ecd2034e11ae6cf688c1c10a4d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 9.
I20260806 20:36:08.975183 15185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59384 (request call id 676) took 2768 ms. Trace:
I20260806 20:36:08.941540 17393 raft_consensus.cc:482] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 92828dd6dbe74f29a04d4e702b502621)
I20260806 20:36:08.941689 17683 raft_consensus.cc:686] T 7f41753baced49a1997940dfc28b2e00 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Leader. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:36:09.003135 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 62149.1us
I20260806 20:36:08.845335 17754 consensus_queue.cc:237] T 7f4b00d6f3484ddbac8e7f8c2bdebba8 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.960297 17663 raft_consensus.cc:3032] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:36:08.991281 14471 rpcz_store.cc:276] 0806 20:36:06.767256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:07.663344 (+896088us) service_pool.cc:227] Handling call
0806 20:36:07.690894 (+ 27550us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:07.867034 (+176140us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:07.867063 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:07.867103 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:36:07.867146 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.990622 (+1123476us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x7b54002af654. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f3f51 00007fb9b34f299e 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86
0806 20:36:08.990669 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:08.990792 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73123,"raft.queue_time_us":69577,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":18610,"spinlock_wait_cycles":171427200}
W20260806 20:36:08.973685 17766 log.cc:924] Time spent T 17eec21790894c559ad9623765626416 P 148bf2a11dba423ab364371f0458e871: Append to log took a long time: real 1.450s user 0.005s sys 0.000s
I20260806 20:36:08.990885 14553 rpcz_store.cc:276] 0806 20:36:07.503614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.963590 (+1459976us) service_pool.cc:227] Handling call
0806 20:36:08.963904 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.990249 (+ 26345us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.990458 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:36:08.997428 17824 log.cc:924] Time spent T b84df1548a8f46c492d5379377b5bc29 P 8e0cfd35f2994f91aaae46c25c6d6bff: Append to log took a long time: real 0.064s user 0.002s sys 0.000s
I20260806 20:36:08.896391 17255 raft_consensus.cc:504] T 08d2a67d79c54f588be9e64dce5b2202 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:08.952849 17376 leader_election.cc:288] T ac95ea9d55914ec6912deffb9a91d8f8 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 8 election: Requested vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:08.968232 14555 raft_consensus.cc:1261] T 91a7667fe05a410bbd313d3d601d18b2 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:36:09.033389 14555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1751) took 1454 ms. Trace:
W20260806 20:36:09.033684 14441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.132:42399, user_credentials={real_user=slave}} blocked reactor thread for 213605us
I20260806 20:36:09.033859 14555 rpcz_store.cc:276] 0806 20:36:07.579036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.967582 (+1388546us) service_pool.cc:227] Handling call
0806 20:36:08.967886 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:09.033112 (+ 65226us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:09.033330 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:08.951102 17688 leader_election.cc:288] T b16d0272c85445ed95a93660f8851656 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:08.956916 17649 consensus_queue.cc:260] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:09.004908 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54498 (request call id 1755) took 1331 ms. Trace:
I20260806 20:36:09.001413 15185 rpcz_store.cc:276] 0806 20:36:06.207056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:06.385266 (+178210us) service_pool.cc:227] Handling call
0806 20:36:08.628574 (+2243308us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:08.665968 (+ 37394us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:08.974947 (+308979us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54006f04d0. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f2a09 00007fb9b84342e9 00007fb9b1b1a684 00007fb9b1b1a612 00007fb9b1b1a541 00007fb9b1b1a4bc 00007fb9b1b19772 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:08.975100 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5034219904}
I20260806 20:36:09.006713 17393 raft_consensus.cc:3037] T fb469b8694714b21bdf2c86de7aa2dad P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:09.027442 14442 leader_election.cc:302] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0cfd35f2994f91aaae46c25c6d6bff, 92828dd6dbe74f29a04d4e702b502621; no voters:
I20260806 20:36:09.037019 14547 rpcz_store.cc:276] 0806 20:36:07.673229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.986826 (+1313597us) service_pool.cc:227] Handling call
0806 20:36:08.987140 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:09.004544 (+ 17404us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:09.004838 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:09.037211 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc270596f078400e9e1b181a9706b516" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d"
W20260806 20:36:09.014748 14444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.248.131:45209 timed out after 1.833s (SENT) [suppressed 19 similar messages]
I20260806 20:36:09.012199 17663 raft_consensus.cc:729] T a187485d6b5e4d9a9a9feafd48522726 P 92828dd6dbe74f29a04d4e702b502621 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
I20260806 20:36:09.051842 14595 leader_election.cc:302] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92828dd6dbe74f29a04d4e702b502621, df1245ecd2034e11ae6cf688c1c10a4d; no voters: 8e0cfd35f2994f91aaae46c25c6d6bff
I20260806 20:36:09.057360 14629 raft_consensus.cc:1261] T 7c10175264634fcd88e06ab41ee2a25c P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:36:09.066462 14631 raft_consensus.cc:2454] T c7a9c839b021436dac626c7630f18f8e P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92828dd6dbe74f29a04d4e702b502621 in term 8.
I20260806 20:36:09.067559 14555 raft_consensus.cc:1261] T d8933c210fdc40cab4369e6aa56f7020 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:36:09.070012 14628 raft_consensus.cc:3037] T dc270596f078400e9e1b181a9706b516 P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:36:09.037931 17661 raft_consensus.cc:2726] T f38960b1dd9245a8ba85fea601b09e94 P df1245ecd2034e11ae6cf688c1c10a4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:36:09.075273 14547 raft_consensus.cc:1261] T 6d7dc7fd91be46ecb624803a9b029e5b P 8e0cfd35f2994f91aaae46c25c6d6bff [term 6 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 20:36:09.076530 14554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54508 (request call id 1850) took 1364 ms. Trace:
I20260806 20:36:09.077696 14474 raft_consensus.cc:3032] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:36:09.078353 14474 raft_consensus.cc:729] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 92828dd6dbe74f29a04d4e702b502621, State: Running, Role: LEADER
W20260806 20:36:09.081826 14442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 54472.3us
I20260806 20:36:09.029459 17459 consensus_queue.cc:1046] T d8933c210fdc40cab4369e6aa56f7020 P 92828dd6dbe74f29a04d4e702b502621 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260806 20:36:09.081038 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ced25ce9f1484081a8671550489f9b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:09.061204 17649 raft_consensus.cc:3037] T 119ce96f4f7f49cfa853221635714edc P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:36:09.087055 14518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.129:44789, user_credentials={real_user=slave}} blocked reactor thread for 95086.9us
I20260806 20:36:09.070820 17688 raft_consensus.cc:2781] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:36:09.083560 17790 raft_consensus.cc:2781] T d01e9209cd924024ae8e7e630b5accd3 P 92828dd6dbe74f29a04d4e702b502621 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:36:09.093139 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a0e4079ebe47aaa3c275ed7f559ab8" candidate_uuid: "148bf2a11dba423ab364371f0458e871" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:09.098376 15185 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P 148bf2a11dba423ab364371f0458e871 [term 8 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:36:09.098462 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb32d3aa416443529139823cf8e2d00b" candidate_uuid: "92828dd6dbe74f29a04d4e702b502621" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" is_pre_election: true
I20260806 20:36:09.101341 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871" is_pre_election: true
I20260806 20:36:09.072616 17602 leader_election.cc:288] T b84df1548a8f46c492d5379377b5bc29 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 6 election: Requested vote from peers 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545), 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399)
I20260806 20:36:09.099680 14478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
W20260806 20:36:09.104996 14444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.248.131:45209, user_credentials={real_user=slave}} blocked reactor thread for 90270.2us
I20260806 20:36:09.109663 14547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36504 (request call id 440) took 1011 ms. Trace:
I20260806 20:36:09.110037 14547 rpcz_store.cc:276] 0806 20:36:08.098184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:09.075903 (+977719us) service_pool.cc:227] Handling call
0806 20:36:09.091459 (+ 15556us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:36:09.109327 (+ 17868us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:36:09.109361 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:36:09.109402 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:36:09.109441 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:36:09.109510 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:36:09.109610 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:36:09.086987 17703 raft_consensus.cc:2781] T c7a9c839b021436dac626c7630f18f8e P 92828dd6dbe74f29a04d4e702b502621 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:36:09.073549 17376 consensus_queue.cc:1035] T f5277702c39f453cb1063cf506183484 P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260806 20:36:09.116124 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621" is_pre_election: true
I20260806 20:36:09.120239 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac95ea9d55914ec6912deffb9a91d8f8" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:09.121905 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:09.123319 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16d0272c85445ed95a93660f8851656" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" is_pre_election: true
I20260806 20:36:09.124015 14555 raft_consensus.cc:1799] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Handling vote request from an unknown peer df1245ecd2034e11ae6cf688c1c10a4d
I20260806 20:36:09.113333 17661 raft_consensus.cc:482] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:36:09.125182 17332 consensus_meta.cc:287] T 7f41753baced49a1997940dfc28b2e00 P df1245ecd2034e11ae6cf688c1c10a4d: Time spent flushing consensus metadata: real 1.023s user 0.008s sys 0.004s
I20260806 20:36:09.124943 17661 raft_consensus.cc:504] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 } } peers { permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 } } peers { permanent_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" member_type: VOTER last_known_addr { host: "127.12.248.131" port: 45209 } }
I20260806 20:36:09.126289 17810 consensus_queue.cc:1046] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "92828dd6dbe74f29a04d4e702b502621" member_type: VOTER last_known_addr { host: "127.12.248.129" port: 44789 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.053s
I20260806 20:36:09.127218 17827 consensus_queue.cc:1035] T d8933c210fdc40cab4369e6aa56f7020 P 148bf2a11dba423ab364371f0458e871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff" member_type: VOTER last_known_addr { host: "127.12.248.130" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20260806 20:36:09.127959 14441 leader_election.cc:398] T b16d0272c85445ed95a93660f8851656 P 92828dd6dbe74f29a04d4e702b502621 [CANDIDATE]: Term 4 election: Vote denied by peer 148bf2a11dba423ab364371f0458e871 (127.12.248.132:42399) with higher term. Message: Invalid argument: T b16d0272c85445ed95a93660f8851656 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 92828dd6dbe74f29a04d4e702b502621 for earlier term 4. Current term is 5.
I20260806 20:36:09.128499 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38500 (request call id 1673) took 1691 ms. Trace:
I20260806 20:36:09.129115 14555 raft_consensus.cc:2379] T b16d0272c85445ed95a93660f8851656 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df1245ecd2034e11ae6cf688c1c10a4d in current term 5: Already voted for candidate 148bf2a11dba423ab364371f0458e871 in this term.
I20260806 20:36:09.128989 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "92828dd6dbe74f29a04d4e702b502621"
I20260806 20:36:09.091415 17688 raft_consensus.cc:482] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8e0cfd35f2994f91aaae46c25c6d6bff)
I20260806 20:36:09.118844 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ca0af1acba4a7ea01bf15f5f887457" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8e0cfd35f2994f91aaae46c25c6d6bff"
I20260806 20:36:09.135288 14518 leader_election.cc:398] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Vote denied by peer 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789) with higher term. Message: Invalid argument: T 7c10175264634fcd88e06ab41ee2a25c P 92828dd6dbe74f29a04d4e702b502621 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0cfd35f2994f91aaae46c25c6d6bff for earlier term 5. Current term is 6.
I20260806 20:36:09.136965 14518 leader_election.cc:401] T 7c10175264634fcd88e06ab41ee2a25c P 8e0cfd35f2994f91aaae46c25c6d6bff [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:36:09.128949 14630 rpcz_store.cc:276] 0806 20:36:07.439317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:36:08.381779 (+942462us) service_pool.cc:227] Handling call
0806 20:36:09.128235 (+746456us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b540006fed4. stack: 00007fb9b0599033 00007fb9b0598f30 00007fb9afef7534 00007fb9b9c038b4 00007fb9b9c03849 00007fb9b3478471 00007fb9b34f72ae 00007fb9b8434c27 00007fb9b1b1cc04 00007fb9b1b1cb92 00007fb9b1b1cac1 00007fb9b1b1ca3c 00007fb9b1b1bcf2 00007fb9b191d94c 00007fb9b191cd86 00007fb9b191c72f
0806 20:36:09.128387 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1126642048}
I20260806 20:36:09.139127 14631 raft_consensus.cc:1261] T 7815d5557ccf4e569039960d0a8f24ba P df1245ecd2034e11ae6cf688c1c10a4d [term 8 FOLLOWER]: Refusing update from remote peer 92828dd6dbe74f29a04d4e702b502621: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:36:09.136157 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84df1548a8f46c492d5379377b5bc29" candidate_uuid: "df1245ecd2034e11ae6cf688c1c10a4d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "148bf2a11dba423ab364371f0458e871"
I20260806 20:36:09.141350 15184 raft_consensus.cc:3037] T b84df1548a8f46c492d5379377b5bc29 P 148bf2a11dba423ab364371f0458e871 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:36:09.144903 14555 raft_consensus.cc:1261] T 17eec21790894c559ad9623765626416 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer 148bf2a11dba423ab364371f0458e871: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:36:09.154433 17816 raft_consensus.cc:2726] T 370a97aea0534403aeb11fe85b5a1ca9 P 8e0cfd35f2994f91aaae46c25c6d6bff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:36:09.155846 14596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.248.130:39545, user_credentials={real_user=slave}} blocked reactor thread for 57978.5us
I20260806 20:36:09.127209 17661 leader_election.cc:288] T 91a7667fe05a410bbd313d3d601d18b2 P df1245ecd2034e11ae6cf688c1c10a4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 92828dd6dbe74f29a04d4e702b502621 (127.12.248.129:44789), 8e0cfd35f2994f91aaae46c25c6d6bff (127.12.248.130:39545)
I20260806 20:36:09.161373 14547 raft_consensus.cc:1261] T 67e455c79bfc43e9ac867cf6719e289c P 8e0cfd35f2994f91aaae46c25c6d6bff [term 7 FOLLOWER]: Refusing update from remote peer df1245ecd2034e11ae6cf688c1c10a4d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:36:09.138772 17688 raft_consensus.cc:3037] T bef4a42f075d4ee1ac14ba3159109a18 P df1245ecd2034e11ae6cf688c1c10a4d [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:36:09.148826 17810 consensus_queue.cc:1225] T 457a5ba876d14681a3fa2338fd9f6c4e P df1245ecd2034e11ae6cf688c1c10a4d [LEADER]: Peer 92828dd6dbe74f29a04d4e702b502621 log is divergent from this leader: its last log entry 8.9 is not in this leader'